[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:09 dist-test-slave-cgwx krb5kdc[4013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:09 dist-test-slave-cgwx krb5kdc[4013](info): set up 2 sockets
Jun 19 16:46:09 dist-test-slave-cgwx krb5kdc[4013](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 16:46:10.603993 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:10 dist-test-slave-cgwx krb5kdc[4013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:10.728369 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34267
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:10.838462 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:10.838786 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:10.838869 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:10.842952 4026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:10.843052 4026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:10.843089 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:10.843128 4026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:10.843168 4026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:10.848474 4026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:10.849601 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:10.850648 4026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:46:10.859527 4026 server_base.cc:1034] running on GCE node
W20260619 16:46:10.859490 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:10.859591 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:10.859477 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:10.860314 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:10.861506 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:10.862739 4026 hybrid_clock.cc:648] HybridClock initialized: now 1781887570862728 us; error 33 us; skew 500 ppm
Jun 19 16:46:10 dist-test-slave-cgwx krb5kdc[4013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887570, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:10.865736 4026 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:10.866637 4026 webserver.cc:460] Webserver started at http://127.3.233.190:42961/ using document root <none> and password file <none>
I20260619 16:46:10.867267 4026 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:10.867359 4026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:10.867660 4026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:10.869915 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0"
format_stamp: "Formatted at 2026-06-19 16:46:10 on dist-test-slave-cgwx"
server_key: "4d9125509b53281c8df43ffd4b45a102"
server_key_iv: "9fa5894cad2d424d2c567493f7e9c62b"
server_key_version: "encryptionkey@0"
I20260619 16:46:10.870524 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0"
format_stamp: "Formatted at 2026-06-19 16:46:10 on dist-test-slave-cgwx"
server_key: "4d9125509b53281c8df43ffd4b45a102"
server_key_iv: "9fa5894cad2d424d2c567493f7e9c62b"
server_key_version: "encryptionkey@0"
I20260619 16:46:10.874533 4026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 16:46:10.877075 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:10.878474 4026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:10.878646 4026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0"
format_stamp: "Formatted at 2026-06-19 16:46:10 on dist-test-slave-cgwx"
server_key: "4d9125509b53281c8df43ffd4b45a102"
server_key_iv: "9fa5894cad2d424d2c567493f7e9c62b"
server_key_version: "encryptionkey@0"
I20260619 16:46:10.878779 4026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:10.899971 4026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:10.903645 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:10.903877 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:10.911335 4026 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34267
I20260619 16:46:10.911353 4093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34267 every 8 connection(s)
I20260619 16:46:10.912346 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:10.915213 4094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:10.915287 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4026
I20260619 16:46:10.915489 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:10.916769 4006 external_mini_cluster.cc:1428] Setting key 67bb0f7ab1790236a7de15d7616f8b28
I20260619 16:46:10.921137 4094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0: Bootstrap starting.
I20260619 16:46:10.923424 4094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:10.924258 4094 log.cc:824] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:10.926383 4094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0: No bootstrap required, opened a new log
I20260619 16:46:10.930377 4094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34267 } }
I20260619 16:46:10.930596 4094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:10.930632 4094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf06759fe7b4fe0abe1a0b298de9fe0, State: Initialized, Role: FOLLOWER
I20260619 16:46:10.931154 4094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34267 } }
I20260619 16:46:10.931280 4094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:10.931366 4094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:10.931485 4094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:10.932427 4094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34267 } }
I20260619 16:46:10.932773 4094 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf06759fe7b4fe0abe1a0b298de9fe0; no voters:
I20260619 16:46:10.933122 4094 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:10.933233 4099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:10.933497 4099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf06759fe7b4fe0abe1a0b298de9fe0, State: Running, Role: LEADER
I20260619 16:46:10.933908 4099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34267 } }
I20260619 16:46:10.934404 4094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:10.935444 4101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf06759fe7b4fe0abe1a0b298de9fe0. Latest consensus state: current_term: 1 leader_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34267 } } }
I20260619 16:46:10.935586 4101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:10.935915 4100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf06759fe7b4fe0abe1a0b298de9fe0" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34267 } } }
I20260619 16:46:10.936045 4100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0 [sys.catalog]: This master's current role is: LEADER
Jun 19 16:46:10 dist-test-slave-cgwx krb5kdc[4013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:10.936221 4108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:10.939991 4108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:10.944231 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:10.919929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.3.233.190:34267)
0619 16:46:10.920433 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:10.920453 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:10.921494 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 16:46:10.924843 (+ 3349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:10.924861 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:10.925250 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:10.925609 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:10.927803 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.929376 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.930811 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:10.931658 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:10.931847 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:10.937393 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:10.937400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:10.937420 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:10.937461 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:10.939931 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.940918 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.940924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.940929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.941093 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:10.941582 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:10.941586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:10.941588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:10.942155 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:10.942393 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:10.943056 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 16:46:10.943330 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":164,"threads_started":1}
I20260619 16:46:10.951023 4108 catalog_manager.cc:1347] Generated new cluster ID: 6774b8c705424d7abb4a362870ebd43e
I20260619 16:46:10.951110 4108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:10.966178 4108 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:10.967589 4108 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:10.978976 4108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf06759fe7b4fe0abe1a0b298de9fe0: Generated new TSK 0
I20260619 16:46:10.980103 4108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:11.017755 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34267
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:11.142903 4120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:11.143272 4120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:11.143353 4120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:11.147744 4120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:11.147838 4120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:11.147935 4120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:11.153646 4120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34267
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:11.154843 4120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:11.156273 4120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:11.166378 4125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:11.166388 4128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:11.166524 4120 server_base.cc:1034] running on GCE node
W20260619 16:46:11.166389 4126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:11.167220 4120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:11.167928 4120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:11.169157 4120 hybrid_clock.cc:648] HybridClock initialized: now 1781887571169107 us; error 56 us; skew 500 ppm
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:11.172374 4120 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:11.173385 4120 webserver.cc:460] Webserver started at http://127.3.233.129:36015/ using document root <none> and password file <none>
I20260619 16:46:11.174050 4120 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:11.174150 4120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:11.174453 4120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:11.176779 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "8a755c658c9a4b35a1af299082d39773"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "dc03687a08ca4a2a8b9d89ac15709f9b"
server_key_iv: "dbf524871524c052aed681f77551d76d"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.177410 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "8a755c658c9a4b35a1af299082d39773"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "dc03687a08ca4a2a8b9d89ac15709f9b"
server_key_iv: "dbf524871524c052aed681f77551d76d"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.181542 4120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 16:46:11.184441 4135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.185911 4120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:46:11.186021 4120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "8a755c658c9a4b35a1af299082d39773"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "dc03687a08ca4a2a8b9d89ac15709f9b"
server_key_iv: "dbf524871524c052aed681f77551d76d"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.186117 4120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:11.211601 4120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:11.215487 4120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:11.215659 4120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:11.216250 4120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:11.217121 4120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:11.217175 4120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.217217 4120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:11.217232 4120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.226873 4120 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:39109
I20260619 16:46:11.226891 4248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:39109 every 8 connection(s)
I20260619 16:46:11.227903 4120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:11.235519 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4120
I20260619 16:46:11.235654 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:11.235942 4006 external_mini_cluster.cc:1428] Setting key f629425022e06000a1b7a3863f5ab5b1
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:11.242906 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.229821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:40387 (local address 127.3.233.190:34267)
0619 16:46:11.229959 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.229965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.230760 (+ 795us) server_negotiation.cc:407] Connection header received
0619 16:46:11.232259 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.232262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.232324 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.232412 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.233617 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.234227 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.235824 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.236137 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.236232 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.239231 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.239236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.239238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.239273 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.240792 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.241330 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.241333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.241335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.241417 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.241770 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.241773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.241775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.241964 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.242077 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.242604 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.242741 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:46:11.243784 4251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.230139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34267 (local address 127.3.233.129:40387)
0619 16:46:11.230599 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.230643 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.232037 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.232591 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.232598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.232967 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:11.233463 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.233476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.234358 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.234366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.235638 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.235650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.236308 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.236313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.236553 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.237233 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:11.237249 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:11.239064 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:11.240918 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:11.240924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:11.240942 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:11.241219 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:11.241511 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:11.241514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:11.241516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:11.241668 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:11.242078 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:11.242086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:11.242359 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.242560 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.242905 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:11.245106 4249 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34267
I20260619 16:46:11.245379 4249 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:11.245833 4249 heartbeater.cc:511] Master 127.3.233.190:34267 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:11.247491 4058 ts_manager.cc:194] Registered new tserver with Master: 8a755c658c9a4b35a1af299082d39773 (127.3.233.129:39109)
I20260619 16:46:11.248863 4058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:40387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:11.262691 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34267
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:11.374046 4254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:11.374318 4254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:11.374363 4254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:11.378712 4254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:11.378777 4254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:11.378871 4254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:11.384446 4254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34267
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:11.385537 4254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:11.386543 4254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:11.396275 4260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:11.396298 4259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:11.396279 4262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:11.396509 4254 server_base.cc:1034] running on GCE node
I20260619 16:46:11.397101 4254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:11.397866 4254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:11.399021 4254 hybrid_clock.cc:648] HybridClock initialized: now 1781887571398996 us; error 34 us; skew 500 ppm
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:11.401919 4254 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:11.402769 4254 webserver.cc:460] Webserver started at http://127.3.233.130:38647/ using document root <none> and password file <none>
I20260619 16:46:11.403359 4254 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:11.403420 4254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:11.403578 4254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:11.405576 4254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "7ca5404a1d1140dd826ab785fbda7c13"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "252ac0a3b7906dcfb663efbbc7fca04c"
server_key_iv: "0b2bd7ca11c6662f037ede68c137ff26"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.406066 4254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "7ca5404a1d1140dd826ab785fbda7c13"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "252ac0a3b7906dcfb663efbbc7fca04c"
server_key_iv: "0b2bd7ca11c6662f037ede68c137ff26"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.409917 4254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:46:11.412530 4269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.413766 4254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:11.413866 4254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "7ca5404a1d1140dd826ab785fbda7c13"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "252ac0a3b7906dcfb663efbbc7fca04c"
server_key_iv: "0b2bd7ca11c6662f037ede68c137ff26"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.414007 4254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:11.425555 4254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:11.428908 4254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:11.429056 4254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:11.429550 4254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:11.430311 4254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:11.430361 4254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.430399 4254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:11.430414 4254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.439817 4254 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:40257
I20260619 16:46:11.439909 4382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:40257 every 8 connection(s)
I20260619 16:46:11.440757 4254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:11.449095 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4254
I20260619 16:46:11.449180 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:11.449502 4006 external_mini_cluster.cc:1428] Setting key 0f00ea899dba47e59c49c591edd68a66
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:11.455722 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.442818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:44897 (local address 127.3.233.190:34267)
0619 16:46:11.443069 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.443076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.443897 (+ 821us) server_negotiation.cc:407] Connection header received
0619 16:46:11.445562 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.445565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.445621 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.445729 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.446880 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.447466 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.448614 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.448839 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.448958 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.451977 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.451981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.451983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.452006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.453639 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.454165 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.454170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.454172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.454257 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.454628 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.454631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.454633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.454811 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.454918 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.455444 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.455565 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260619 16:46:11.456646 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.443265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34267 (local address 127.3.233.130:44897)
0619 16:46:11.443714 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.443764 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.445286 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.445898 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.445905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.446252 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:11.446730 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.446742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.447595 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.447604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.448490 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.448499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.448995 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.448999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.449213 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.449851 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:11.449870 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:11.451780 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:11.453764 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:11.453771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:11.453786 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:11.454059 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:11.454357 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:11.454360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:11.454362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:11.454520 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:11.454941 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:11.454951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:11.455204 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.455396 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.455724 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260619 16:46:11.457801 4383 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34267
I20260619 16:46:11.458056 4383 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:11.458541 4383 heartbeater.cc:511] Master 127.3.233.190:34267 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:46:11.459703 4058 ts_manager.cc:194] Registered new tserver with Master: 7ca5404a1d1140dd826ab785fbda7c13 (127.3.233.130:40257)
I20260619 16:46:11.460234 4058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:44897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:11.477872 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34267
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:11.591435 4388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:11.591719 4388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:11.591765 4388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:11.596005 4388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:11.596072 4388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:11.596135 4388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:11.601507 4388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34267
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:11.602658 4388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:11.603854 4388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:11.614010 4394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:11.613961 4396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:11.614080 4388 server_base.cc:1034] running on GCE node
W20260619 16:46:11.613950 4393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:11.614748 4388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:11.615489 4388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:11.616668 4388 hybrid_clock.cc:648] HybridClock initialized: now 1781887571616644 us; error 47 us; skew 500 ppm
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:11.619894 4388 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:11.620926 4388 webserver.cc:460] Webserver started at http://127.3.233.131:41543/ using document root <none> and password file <none>
I20260619 16:46:11.621542 4388 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:11.621639 4388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:11.621881 4388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:11.624145 4388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "902aaa7cd6224e6b90fee5c8ee6d846a"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "75425dba90fc512e295f6e117a11df8f"
server_key_iv: "5f65e0a7b2b82abb19ed9e2719269614"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.624734 4388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "902aaa7cd6224e6b90fee5c8ee6d846a"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "75425dba90fc512e295f6e117a11df8f"
server_key_iv: "5f65e0a7b2b82abb19ed9e2719269614"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.628774 4388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:11.631441 4403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.632776 4388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:11.632923 4388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "902aaa7cd6224e6b90fee5c8ee6d846a"
format_stamp: "Formatted at 2026-06-19 16:46:11 on dist-test-slave-cgwx"
server_key: "75425dba90fc512e295f6e117a11df8f"
server_key_iv: "5f65e0a7b2b82abb19ed9e2719269614"
server_key_version: "encryptionkey@0"
I20260619 16:46:11.633050 4388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:11.651091 4388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:11.654639 4388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:11.654798 4388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:11.655439 4388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:11.656240 4388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:11.656284 4388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.656324 4388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:11.656339 4388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:11.665303 4388 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:38117
I20260619 16:46:11.665422 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:38117 every 8 connection(s)
I20260619 16:46:11.666252 4388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:11.674474 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4388
I20260619 16:46:11.674609 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:11.674898 4006 external_mini_cluster.cc:1428] Setting key 5f687790bad67b040375443b503bf5a5
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:11.680960 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.668255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:52071 (local address 127.3.233.190:34267)
0619 16:46:11.668456 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.668461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.669479 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 16:46:11.670860 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.670862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.670911 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.671065 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.672334 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.672867 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.674013 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.674264 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.674346 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.677618 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.677623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.677626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.677657 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.679196 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.679658 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.679661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.679662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.679733 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.680043 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.680045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.680046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.680208 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.680280 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.680753 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.680847 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:46:11.681814 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.668757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34267 (local address 127.3.233.131:52071)
0619 16:46:11.669251 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:11.669297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:11.670646 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:11.671237 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:11.671245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:11.671667 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:11.672159 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.672173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.673001 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.673004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.673890 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:11.673898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.674510 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.674514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:11.674745 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.675405 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:11.675424 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:11.677444 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:11.679296 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:11.679301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:11.679314 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:11.679572 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:11.679812 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:11.679814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:11.679815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:11.679965 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:11.680289 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:11.680294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:11.680544 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 16:46:11.680701 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 16:46:11.680992 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":120,"threads_started":1}
I20260619 16:46:11.683221 4517 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34267
I20260619 16:46:11.683499 4517 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:11.684087 4517 heartbeater.cc:511] Master 127.3.233.190:34267 requested a full tablet report, sending...
I20260619 16:46:11.685125 4058 ts_manager.cc:194] Registered new tserver with Master: 902aaa7cd6224e6b90fee5c8ee6d846a (127.3.233.131:38117)
I20260619 16:46:11.685762 4058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:52071
I20260619 16:46:11.689713 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:11.716359 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.705872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.3.233.129:39109)
0619 16:46:11.706233 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.706247 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.706275 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:46:11.706343 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.706350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.706586 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.706732 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.707477 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.708290 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.709264 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.709729 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.709823 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.712622 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.712632 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.712646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.712673 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.714393 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.714980 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.715003 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.715009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.715100 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.715452 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.715454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.715456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.715714 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.715881 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.716084 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.716217 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
W20260619 16:46:11.717543 4163 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44832
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:11.744333 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:11.735936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.3.233.129:39109)
0619 16:46:11.736109 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:11.736114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:11.736285 (+ 171us) server_negotiation.cc:407] Connection header received
0619 16:46:11.736485 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:11.736489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:11.736532 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:11.736618 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:11.737325 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.737879 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.738818 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:11.739113 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:11.739223 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:11.741592 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:11.741595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:11.741597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:11.741623 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:11.742895 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.743326 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.743328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.743329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.743402 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:11.743664 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:11.743665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:11.743667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:11.743817 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:11.743927 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:11.744113 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:46:11.744212 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:46:11.745431 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4120
I20260619 16:46:11.751451 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4254
I20260619 16:46:11.757009 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4388
I20260619 16:46:11.762475 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4026
2026-06-19T16:46:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2241 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4531](info): set up 2 sockets
Jun 19 16:46:11 dist-test-slave-cgwx krb5kdc[4531](info): commencing operation
krb5kdc: starting...
W20260619 16:46:12.855077 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:12 dist-test-slave-cgwx krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:12.978914 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:46011
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:46011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:13.104799 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:13.105082 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:13.105122 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:13.109166 4544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:13.109232 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:13.109252 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:13.109269 4544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:13.109284 4544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:13.114764 4544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:46011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:46011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:13.115903 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:13.116919 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:46:13.125576 4544 server_base.cc:1034] running on GCE node
W20260619 16:46:13.125517 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:13.125525 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:13.125556 4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:13.126271 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:13.127465 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:13.128651 4544 hybrid_clock.cc:648] HybridClock initialized: now 1781887573128610 us; error 47 us; skew 500 ppm
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:13.131647 4544 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:13.132472 4544 webserver.cc:460] Webserver started at http://127.3.233.190:42605/ using document root <none> and password file <none>
I20260619 16:46:13.132988 4544 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:13.133044 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:13.133201 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:13.135305 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "cbae25e024284b2ba34ebb0ee9982a6f"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "d958f42c0726215e64c011aeb0f82d10"
server_key_iv: "3e4aa765f2f0f64fdcc5fe6535478e64"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.135828 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "cbae25e024284b2ba34ebb0ee9982a6f"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "d958f42c0726215e64c011aeb0f82d10"
server_key_iv: "3e4aa765f2f0f64fdcc5fe6535478e64"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.139500 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:46:13.141884 4559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.143373 4544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:13.143479 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "cbae25e024284b2ba34ebb0ee9982a6f"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "d958f42c0726215e64c011aeb0f82d10"
server_key_iv: "3e4aa765f2f0f64fdcc5fe6535478e64"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.143571 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:13.157205 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:13.160719 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:13.160892 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:13.168164 4544 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:46011
I20260619 16:46:13.168256 4611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:46011 every 8 connection(s)
I20260619 16:46:13.169198 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:13.172151 4612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:13.175276 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4544
I20260619 16:46:13.175371 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:13.175647 4006 external_mini_cluster.cc:1428] Setting key f372de062d0c0b744eea3b849ad2073a
I20260619 16:46:13.178256 4612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f: Bootstrap starting.
I20260619 16:46:13.180895 4612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:13.181869 4612 log.cc:824] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:13.184355 4612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f: No bootstrap required, opened a new log
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:13.188755 4612 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46011 } }
I20260619 16:46:13.189057 4612 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:13.189119 4612 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbae25e024284b2ba34ebb0ee9982a6f, State: Initialized, Role: FOLLOWER
I20260619 16:46:13.189631 4612 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46011 } }
I20260619 16:46:13.189792 4612 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:13.189863 4612 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:13.189963 4612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:13.190913 4612 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46011 } }
I20260619 16:46:13.191349 4612 leader_election.cc:302] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbae25e024284b2ba34ebb0ee9982a6f; no voters:
I20260619 16:46:13.191679 4612 leader_election.cc:288] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:13.191792 4617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:13.191988 4617 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [term 1 LEADER]: Becoming Leader. State: Replica: cbae25e024284b2ba34ebb0ee9982a6f, State: Running, Role: LEADER
I20260619 16:46:13.192279 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.177209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.3.233.190:46011)
0619 16:46:13.177625 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.177641 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.177678 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:46:13.178307 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.178323 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.178729 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.179105 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:13.179980 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.180970 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.182197 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.182700 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.182819 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.185768 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:13.185775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:13.185796 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:13.185832 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:13.188602 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.189220 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.189229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.189239 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.189375 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.189680 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.189684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.189686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.190261 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:13.190484 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.190865 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.191268 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":142,"threads_started":1}
I20260619 16:46:13.192299 4617 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46011 } }
I20260619 16:46:13.192981 4612 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:13.193524 4617 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbae25e024284b2ba34ebb0ee9982a6f. Latest consensus state: current_term: 1 leader_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46011 } } }
I20260619 16:46:13.193645 4617 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:13.195297 4618 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbae25e024284b2ba34ebb0ee9982a6f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46011 } } }
I20260619 16:46:13.196022 4618 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f [sys.catalog]: This master's current role is: LEADER
W20260619 16:46:13.197903 4632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:13.198009 4632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:13.198084 4621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:13.198755 4621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:13.204982 4621 catalog_manager.cc:1347] Generated new cluster ID: a2bff30ace604bd48b85f78a9e390eef
I20260619 16:46:13.205063 4621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:13.226764 4621 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:13.227797 4621 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:13.232106 4621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbae25e024284b2ba34ebb0ee9982a6f: Generated new TSK 0
I20260619 16:46:13.233085 4621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:13.261674 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46011
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:13.375260 4638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:13.375528 4638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:13.375573 4638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:13.379714 4638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:13.379777 4638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:13.379839 4638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:13.385267 4638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:13.386397 4638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:13.387621 4638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:13.397496 4643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:13.397612 4644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:13.397512 4646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:13.397742 4638 server_base.cc:1034] running on GCE node
I20260619 16:46:13.398280 4638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:13.398969 4638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:13.400166 4638 hybrid_clock.cc:648] HybridClock initialized: now 1781887573400132 us; error 48 us; skew 500 ppm
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:13.403152 4638 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:13.404011 4638 webserver.cc:460] Webserver started at http://127.3.233.129:45261/ using document root <none> and password file <none>
I20260619 16:46:13.404582 4638 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:13.404675 4638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:13.404922 4638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:13.407070 4638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "e0cb154aeda442b9b4d114178bdfc160"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "e2b8744bd258cb5bddc7e41f34b70b0b"
server_key_iv: "daffd93592dc96462ea0a1b42c6d3df1"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.407660 4638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "e0cb154aeda442b9b4d114178bdfc160"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "e2b8744bd258cb5bddc7e41f34b70b0b"
server_key_iv: "daffd93592dc96462ea0a1b42c6d3df1"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.411684 4638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:13.414490 4653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.415877 4638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:13.416020 4638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "e0cb154aeda442b9b4d114178bdfc160"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "e2b8744bd258cb5bddc7e41f34b70b0b"
server_key_iv: "daffd93592dc96462ea0a1b42c6d3df1"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.416142 4638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:13.448247 4638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:13.451534 4638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:13.451745 4638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:13.452335 4638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:13.453217 4638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:13.453285 4638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.453356 4638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:13.453406 4638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.462810 4638 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:43739
I20260619 16:46:13.462827 4766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:43739 every 8 connection(s)
I20260619 16:46:13.463775 4638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:13.468672 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4638
I20260619 16:46:13.468809 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:13.469079 4006 external_mini_cluster.cc:1428] Setting key c8925e61f872e171f7edce351e9d2121
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:13.478720 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.465733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:46999 (local address 127.3.233.190:46011)
0619 16:46:13.465879 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.465883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.466723 (+ 840us) server_negotiation.cc:407] Connection header received
0619 16:46:13.468162 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.468166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.468223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.468306 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:13.469786 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.470336 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.471564 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.471808 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.471922 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.474537 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:13.474540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:13.474543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:13.474569 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:13.476279 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.476937 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.476943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.476945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.477038 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.477421 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.477427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.477429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.477644 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:13.477759 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.478372 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.478528 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:46:13.479704 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.466115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:46011 (local address 127.3.233.129:46999)
0619 16:46:13.466549 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:13.466617 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:13.467927 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:13.468508 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:13.468516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:13.468993 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:13.469621 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.469646 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.470469 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.470472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.471435 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.471445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.471963 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.471967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.472151 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.472814 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:13.472831 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:13.474354 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:13.476477 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:13.476485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:13.476506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:13.476819 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:13.477139 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:13.477144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:13.477146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:13.477314 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:13.477758 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:13.477767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:13.478116 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 16:46:13.478331 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 16:46:13.478697 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:13.481153 4767 heartbeater.cc:347] Connected to a master server at 127.3.233.190:46011
I20260619 16:46:13.481458 4767 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:13.482084 4767 heartbeater.cc:511] Master 127.3.233.190:46011 requested a full tablet report, sending...
I20260619 16:46:13.483914 4576 ts_manager.cc:194] Registered new tserver with Master: e0cb154aeda442b9b4d114178bdfc160 (127.3.233.129:43739)
I20260619 16:46:13.485330 4576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:46999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:13.497674 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46011
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:13.610527 4772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:13.610857 4772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:13.610940 4772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:13.615121 4772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:13.615211 4772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:13.615307 4772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:13.620957 4772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:13.622189 4772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:13.623363 4772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:13.633562 4778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:13.633689 4772 server_base.cc:1034] running on GCE node
W20260619 16:46:13.633611 4777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:13.633633 4780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:13.634398 4772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:13.635174 4772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:13.636375 4772 hybrid_clock.cc:648] HybridClock initialized: now 1781887573636357 us; error 41 us; skew 500 ppm
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:13.639371 4772 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:13.640261 4772 webserver.cc:460] Webserver started at http://127.3.233.130:38231/ using document root <none> and password file <none>
I20260619 16:46:13.640798 4772 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:13.640857 4772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:13.641054 4772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:13.643270 4772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "a9b2c4d1ec8141489c20fe52de74c028"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "e8ddba4cd5c631706957524c5c192164"
server_key_iv: "a0cdddae6b0040815d9cb621e3eae3aa"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.643807 4772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "a9b2c4d1ec8141489c20fe52de74c028"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "e8ddba4cd5c631706957524c5c192164"
server_key_iv: "a0cdddae6b0040815d9cb621e3eae3aa"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.647791 4772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:46:13.650204 4787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.651490 4772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:13.651595 4772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "a9b2c4d1ec8141489c20fe52de74c028"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "e8ddba4cd5c631706957524c5c192164"
server_key_iv: "a0cdddae6b0040815d9cb621e3eae3aa"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.651682 4772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:13.669562 4772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:13.673242 4772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:13.673411 4772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:13.673959 4772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:13.674775 4772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:13.674821 4772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.674861 4772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:13.674877 4772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.683636 4772 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:40381
I20260619 16:46:13.683663 4900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:40381 every 8 connection(s)
I20260619 16:46:13.684549 4772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:13.694244 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4772
I20260619 16:46:13.694357 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:13.694731 4006 external_mini_cluster.cc:1428] Setting key c2f79066ffec1b5a437d786676330b4e
I20260619 16:46:13.699100 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.686339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:48103 (local address 127.3.233.190:46011)
0619 16:46:13.686521 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.686525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.687355 (+ 830us) server_negotiation.cc:407] Connection header received
0619 16:46:13.688675 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.688678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.688723 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.688803 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:13.690081 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.690587 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.691991 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.692208 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.692286 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.695064 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:13.695067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:13.695069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:13.695092 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:13.696810 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.697406 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.697409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.697410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.697480 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.697855 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.697857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.697858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.698016 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:13.698091 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.698727 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.698859 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:46:13.699815 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.686723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:46011 (local address 127.3.233.130:48103)
0619 16:46:13.687181 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:13.687225 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:13.688474 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:13.689049 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:13.689056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:13.689434 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:13.689937 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.689950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.690708 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.690711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.691880 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.691890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.692333 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.692336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.692472 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.693119 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:13.693134 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:13.694864 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:13.696927 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:13.696933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:13.696947 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:13.697304 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:13.697562 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:13.697568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:13.697573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:13.697761 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:13.698111 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:13.698121 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:13.698454 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 16:46:13.698628 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:46:13.698875 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":97,"threads_started":1}
I20260619 16:46:13.701107 4901 heartbeater.cc:347] Connected to a master server at 127.3.233.190:46011
I20260619 16:46:13.701336 4901 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:13.701817 4901 heartbeater.cc:511] Master 127.3.233.190:46011 requested a full tablet report, sending...
I20260619 16:46:13.702889 4576 ts_manager.cc:194] Registered new tserver with Master: a9b2c4d1ec8141489c20fe52de74c028 (127.3.233.130:40381)
I20260619 16:46:13.703540 4576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:48103
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:13.722576 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46011
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:13.833307 4906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:13.833582 4906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:13.833658 4906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:13.837805 4906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:13.837869 4906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:13.837960 4906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:13.843385 4906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:13.844512 4906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:13.845799 4906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:13.855973 4914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:13.855976 4911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:13.856091 4906 server_base.cc:1034] running on GCE node
W20260619 16:46:13.855996 4912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:13.856617 4906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:13.857270 4906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:13.858500 4906 hybrid_clock.cc:648] HybridClock initialized: now 1781887573858476 us; error 33 us; skew 500 ppm
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:13.861685 4906 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:13.862600 4906 webserver.cc:460] Webserver started at http://127.3.233.131:44021/ using document root <none> and password file <none>
I20260619 16:46:13.863232 4906 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:13.863296 4906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:13.863556 4906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:13.865778 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "68fc31b2483044db86c83a75cbb83eaa"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "ec123edd092bc89ca256d2eedaf8182c"
server_key_iv: "27e4e421ae5bca4c126979bc9c4506e7"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.866369 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "68fc31b2483044db86c83a75cbb83eaa"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "ec123edd092bc89ca256d2eedaf8182c"
server_key_iv: "27e4e421ae5bca4c126979bc9c4506e7"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.870222 4906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:46:13.872906 4921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.874218 4906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:46:13.874425 4906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "68fc31b2483044db86c83a75cbb83eaa"
format_stamp: "Formatted at 2026-06-19 16:46:13 on dist-test-slave-cgwx"
server_key: "ec123edd092bc89ca256d2eedaf8182c"
server_key_iv: "27e4e421ae5bca4c126979bc9c4506e7"
server_key_version: "encryptionkey@0"
I20260619 16:46:13.874553 4906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:13.888233 4906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:13.891695 4906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:13.891919 4906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:13.892632 4906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:13.893486 4906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:13.893566 4906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.893676 4906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:13.893723 4906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:13.904100 4906 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:38543
I20260619 16:46:13.904124 5034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:38543 every 8 connection(s)
I20260619 16:46:13.905109 4906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:13.909178 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 4906
I20260619 16:46:13.909332 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:13.909621 4006 external_mini_cluster.cc:1428] Setting key c63814f72301e2b6887cf8c4f0d23206
Jun 19 16:46:13 dist-test-slave-cgwx krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:13.921620 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.907170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:60269 (local address 127.3.233.190:46011)
0619 16:46:13.907315 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.907325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.908337 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 16:46:13.910275 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.910279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.910340 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.910480 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:13.912090 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.912725 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.914209 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.914440 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.914528 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.917466 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:13.917469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:13.917471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:13.917496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:13.919382 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.920021 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.920025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.920027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.920110 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.920460 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.920463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.920465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.920653 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:13.920759 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.921318 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.921448 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:46:13.922631 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.907675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:46011 (local address 127.3.233.131:60269)
0619 16:46:13.908153 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:13.908197 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:13.910072 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:13.910722 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:13.910732 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:13.911283 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:13.911956 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.911975 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.912872 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.912875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.914077 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.914087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.914613 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.914616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.914795 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.915524 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:13.915541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:13.917258 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:13.919534 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:13.919540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:13.919556 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:13.919895 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:13.920199 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:13.920204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:13.920206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:13.920353 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:13.920756 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:13.920764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:13.921070 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 16:46:13.921259 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 16:46:13.921593 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
I20260619 16:46:13.923981 5035 heartbeater.cc:347] Connected to a master server at 127.3.233.190:46011
I20260619 16:46:13.924224 5035 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:13.924731 5035 heartbeater.cc:511] Master 127.3.233.190:46011 requested a full tablet report, sending...
I20260619 16:46:13.925758 4576 ts_manager.cc:194] Registered new tserver with Master: 68fc31b2483044db86c83a75cbb83eaa (127.3.233.131:38543)
I20260619 16:46:13.926333 4576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:60269
I20260619 16:46:13.935607 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:13.946909 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.939046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.3.233.190:46011)
0619 16:46:13.939211 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.939214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.939319 (+ 105us) server_negotiation.cc:407] Connection header received
0619 16:46:13.939418 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.939420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.939493 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.939559 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:13.940318 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.940878 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.941727 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.941948 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.942056 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.943625 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:13.943628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:13.943630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:13.943655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:13.945156 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.945573 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.945576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.945577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.945650 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:13.945924 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:13.945927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:13.945928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:13.946142 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:13.946244 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.946588 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.946725 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:46:13.952658 4576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:13.955827 4576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:13.971647 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.965282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:40381 (local address 127.0.0.1:36272)
0619 16:46:13.966011 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:13.966067 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:13.966202 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:13.967172 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:13.967179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:13.967222 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:13.967368 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.967378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.968688 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.968693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.970061 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.970070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.971024 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.971030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.971159 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.971206 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 16:46:13.971327 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:46:13.971472 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":28,"spinlock_wait_cycles":9984,"thread_start_us":70,"threads_started":1}
I20260619 16:46:13.971658 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.965550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.3.233.130:40381)
0619 16:46:13.966087 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.966103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.966135 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:46:13.966475 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.966484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.966741 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.966892 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:13.967519 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.968543 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.970193 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.970806 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.970965 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.971147 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.971374 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.971482 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":130,"threads_started":1}
I20260619 16:46:13.972651 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.966123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38543 (local address 127.0.0.1:36412)
0619 16:46:13.966783 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:13.966798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:13.966886 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:13.967589 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:13.967599 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:13.967616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:13.967708 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.967715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.969349 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.969353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.970766 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.970777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.972129 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.972132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.972315 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.972333 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:46:13.972404 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:46:13.972482 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":68,"thread_start_us":101,"threads_started":1}
I20260619 16:46:13.972878 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.966280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:43739 (local address 127.0.0.1:60816)
0619 16:46:13.967081 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:13.967121 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:13.967229 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:13.967816 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:13.967820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:13.967840 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:13.967933 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.967939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.969370 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.969373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.970886 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:13.970897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.972119 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.972123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:13.972287 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.972324 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:46:13.972550 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 16:46:13.972615 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":256,"spinlock_wait_cycles":3712,"thread_start_us":99,"threads_started":1}
I20260619 16:46:13.973019 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.966645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.3.233.129:43739)
0619 16:46:13.967303 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.967319 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.967343 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:46:13.967420 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.967428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.967664 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.967827 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:13.968016 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.969233 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.971088 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.971979 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.972094 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.972256 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.972731 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.972842 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":97,"threads_started":1}
I20260619 16:46:13.973320 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:13.965677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.3.233.131:38543)
0619 16:46:13.966363 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:13.966379 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:13.967002 (+ 623us) server_negotiation.cc:407] Connection header received
0619 16:46:13.967190 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:13.967196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:13.967402 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:13.967603 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:13.967845 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.969174 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.970942 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:13.971822 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:13.971941 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:13.972079 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:13.973034 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 16:46:13.973153 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":169,"threads_started":1}
I20260619 16:46:13.975853 4701 tablet_service.cc:1467] Processing CreateTablet for tablet b76a81e753ff4dc1aa2dabe3695d349a (DEFAULT_TABLE table=test-table [id=ff4277ed793441d38b494d9c31e5af8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:13.975870 4835 tablet_service.cc:1467] Processing CreateTablet for tablet b76a81e753ff4dc1aa2dabe3695d349a (DEFAULT_TABLE table=test-table [id=ff4277ed793441d38b494d9c31e5af8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:13.977144 4701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76a81e753ff4dc1aa2dabe3695d349a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:13.977144 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76a81e753ff4dc1aa2dabe3695d349a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:13.977778 4969 tablet_service.cc:1467] Processing CreateTablet for tablet b76a81e753ff4dc1aa2dabe3695d349a (DEFAULT_TABLE table=test-table [id=ff4277ed793441d38b494d9c31e5af8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:13.978869 4969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76a81e753ff4dc1aa2dabe3695d349a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:13.985335 5052 tablet_bootstrap.cc:492] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160: Bootstrap starting.
I20260619 16:46:13.986243 5054 tablet_bootstrap.cc:492] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa: Bootstrap starting.
I20260619 16:46:13.986807 5053 tablet_bootstrap.cc:492] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028: Bootstrap starting.
I20260619 16:46:13.988088 5052 tablet_bootstrap.cc:654] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:13.988890 5052 log.cc:824] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:13.989122 5054 tablet_bootstrap.cc:654] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:13.989653 5053 tablet_bootstrap.cc:654] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:13.990108 5054 log.cc:824] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:13.990553 5053 log.cc:824] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:13.991127 5052 tablet_bootstrap.cc:492] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160: No bootstrap required, opened a new log
I20260619 16:46:13.991353 5052 ts_tablet_manager.cc:1397] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:46:13.992311 5054 tablet_bootstrap.cc:492] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa: No bootstrap required, opened a new log
I20260619 16:46:13.992595 5054 ts_tablet_manager.cc:1397] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:46:13.992625 5053 tablet_bootstrap.cc:492] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028: No bootstrap required, opened a new log
I20260619 16:46:13.992781 5053 ts_tablet_manager.cc:1397] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260619 16:46:13.996239 5052 raft_consensus.cc:348] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:13.996503 5052 raft_consensus.cc:374] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:13.996592 5052 raft_consensus.cc:729] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cb154aeda442b9b4d114178bdfc160, State: Initialized, Role: FOLLOWER
I20260619 16:46:13.996848 5053 raft_consensus.cc:348] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:13.997164 5053 raft_consensus.cc:374] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:13.997009 5054 raft_consensus.cc:348] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:13.997090 5052 consensus_queue.cc:260] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:13.997249 5054 raft_consensus.cc:374] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:13.997282 5053 raft_consensus.cc:729] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b2c4d1ec8141489c20fe52de74c028, State: Initialized, Role: FOLLOWER
I20260619 16:46:13.997336 5054 raft_consensus.cc:729] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fc31b2483044db86c83a75cbb83eaa, State: Initialized, Role: FOLLOWER
I20260619 16:46:13.997756 5053 consensus_queue.cc:260] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:13.997783 5054 consensus_queue.cc:260] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:13.998193 5052 ts_tablet_manager.cc:1428] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:46:13.998492 5035 heartbeater.cc:503] Master 127.3.233.190:46011 was elected leader, sending a full tablet report...
I20260619 16:46:13.998735 4901 heartbeater.cc:503] Master 127.3.233.190:46011 was elected leader, sending a full tablet report...
I20260619 16:46:13.998787 5054 ts_tablet_manager.cc:1428] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:46:13.999251 4767 heartbeater.cc:503] Master 127.3.233.190:46011 was elected leader, sending a full tablet report...
I20260619 16:46:14.003499 5053 ts_tablet_manager.cc:1428] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20260619 16:46:14.027191 5060 raft_consensus.cc:482] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:14.027386 5060 raft_consensus.cc:504] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:14.028524 5060 leader_election.cc:288] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9b2c4d1ec8141489c20fe52de74c028 (127.3.233.130:40381), e0cb154aeda442b9b4d114178bdfc160 (127.3.233.129:43739)
I20260619 16:46:14.033113 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.028774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:40381 (local address 127.3.233.131:59179)
0619 16:46:14.028997 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.029012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:14.029156 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:14.029410 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:14.029413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:14.029434 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:14.029574 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.029582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.030597 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.030600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.031944 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.031955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.032671 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.032677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.032795 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.032817 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:46:14.032897 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:46:14.032953 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 16:46:14.033141 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.028990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:59179 (local address 127.3.233.130:40381)
0619 16:46:14.029156 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.029162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.029184 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:46:14.029247 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.029251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.029303 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.029388 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:14.029701 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.030480 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.032114 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.032538 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.032631 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.032672 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.032933 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.033018 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 16:46:14.033221 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.029268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:37753 (local address 127.3.233.129:43739)
0619 16:46:14.029419 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.029424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.029518 (+ 94us) server_negotiation.cc:407] Connection header received
0619 16:46:14.029599 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.029601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.029670 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.029749 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:14.030020 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.030721 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.032192 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.032643 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.032764 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.032811 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.033023 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.033094 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:46:14.033262 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.029084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:43739 (local address 127.3.233.131:37753)
0619 16:46:14.029418 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:14.029431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:14.029516 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:14.029759 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:14.029765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:14.029787 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:14.029897 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.029905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.030814 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.030817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.032040 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:14.032047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.032774 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.032777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:14.032856 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.032912 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 16:46:14.032988 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:46:14.033050 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":129,"threads_started":1}
I20260619 16:46:14.033880 4721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a81e753ff4dc1aa2dabe3695d349a" candidate_uuid: "68fc31b2483044db86c83a75cbb83eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cb154aeda442b9b4d114178bdfc160" is_pre_election: true
I20260619 16:46:14.033895 4855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a81e753ff4dc1aa2dabe3695d349a" candidate_uuid: "68fc31b2483044db86c83a75cbb83eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b2c4d1ec8141489c20fe52de74c028" is_pre_election: true
I20260619 16:46:14.034183 4855 raft_consensus.cc:2454] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc31b2483044db86c83a75cbb83eaa in term 0.
I20260619 16:46:14.034186 4721 raft_consensus.cc:2454] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc31b2483044db86c83a75cbb83eaa in term 0.
I20260619 16:46:14.034646 4924 leader_election.cc:302] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fc31b2483044db86c83a75cbb83eaa, a9b2c4d1ec8141489c20fe52de74c028; no voters:
I20260619 16:46:14.034950 5060 raft_consensus.cc:2781] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:14.035054 5060 raft_consensus.cc:482] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:14.035112 5060 raft_consensus.cc:3037] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:14.036286 5060 raft_consensus.cc:504] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:14.036629 5060 leader_election.cc:288] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [CANDIDATE]: Term 1 election: Requested vote from peers a9b2c4d1ec8141489c20fe52de74c028 (127.3.233.130:40381), e0cb154aeda442b9b4d114178bdfc160 (127.3.233.129:43739)
I20260619 16:46:14.037005 4855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a81e753ff4dc1aa2dabe3695d349a" candidate_uuid: "68fc31b2483044db86c83a75cbb83eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b2c4d1ec8141489c20fe52de74c028"
I20260619 16:46:14.037138 4855 raft_consensus.cc:3037] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:14.037113 4721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a81e753ff4dc1aa2dabe3695d349a" candidate_uuid: "68fc31b2483044db86c83a75cbb83eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cb154aeda442b9b4d114178bdfc160"
I20260619 16:46:14.037286 4721 raft_consensus.cc:3037] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:14.038235 4855 raft_consensus.cc:2454] T b76a81e753ff4dc1aa2dabe3695d349a P a9b2c4d1ec8141489c20fe52de74c028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc31b2483044db86c83a75cbb83eaa in term 1.
I20260619 16:46:14.038425 4721 raft_consensus.cc:2454] T b76a81e753ff4dc1aa2dabe3695d349a P e0cb154aeda442b9b4d114178bdfc160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc31b2483044db86c83a75cbb83eaa in term 1.
I20260619 16:46:14.038599 4924 leader_election.cc:302] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fc31b2483044db86c83a75cbb83eaa, a9b2c4d1ec8141489c20fe52de74c028; no voters:
I20260619 16:46:14.038961 5060 raft_consensus.cc:2781] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:14.039206 5060 raft_consensus.cc:686] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [term 1 LEADER]: Becoming Leader. State: Replica: 68fc31b2483044db86c83a75cbb83eaa, State: Running, Role: LEADER
I20260619 16:46:14.039551 5060 consensus_queue.cc:237] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } }
I20260619 16:46:14.042876 4576 catalog_manager.cc:5495] T b76a81e753ff4dc1aa2dabe3695d349a P 68fc31b2483044db86c83a75cbb83eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fc31b2483044db86c83a75cbb83eaa (127.3.233.131). New cstate: current_term: 1 leader_uuid: "68fc31b2483044db86c83a75cbb83eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9b2c4d1ec8141489c20fe52de74c028" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 40381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fc31b2483044db86c83a75cbb83eaa" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0cb154aeda442b9b4d114178bdfc160" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43739 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:14.059188 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.049272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.3.233.129:43739)
0619 16:46:14.049442 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.049448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.049655 (+ 207us) server_negotiation.cc:407] Connection header received
0619 16:46:14.049829 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.049832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.049908 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.050008 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:14.050778 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.051434 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.052361 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.052609 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.052729 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.055522 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:14.055534 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:14.055548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:14.055573 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:14.057308 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.057778 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.057783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.057784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.057862 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.058159 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.058164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.058166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.058419 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:14.058645 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.058796 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.058965 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:14.087623 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.078380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.3.233.129:43739)
0619 16:46:14.078503 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.078506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.078865 (+ 359us) server_negotiation.cc:407] Connection header received
0619 16:46:14.079084 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.079087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.079138 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.079251 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:14.079999 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.080579 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.081405 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.081639 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.081726 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.084241 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:14.084244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:14.084246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:14.084272 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:14.085877 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.086294 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.086297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.086298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.086393 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.086681 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.086684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.086685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.086860 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:14.087083 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.087303 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.087412 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:14.114800 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:14.105786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.3.233.129:43739)
0619 16:46:14.106017 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:14.106022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:14.106135 (+ 113us) server_negotiation.cc:407] Connection header received
0619 16:46:14.106309 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:14.106313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:14.106365 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:14.106468 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:14.107238 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.107740 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.108589 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:14.108804 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:14.108932 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:14.111561 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:14.111572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:14.111575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:14.111598 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:14.113074 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.113516 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.113521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.113523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.113604 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:14.113901 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:14.113906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:14.113907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:14.114082 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:14.114222 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:14.114507 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 16:46:14.114635 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260619 16:46:14.116220 4681 tablet_service.cc:2908] Rejecting scan request for tablet b76a81e753ff4dc1aa2dabe3695d349a: Uninitialized: clean time has not yet been initialized
I20260619 16:46:14.118346 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4638
I20260619 16:46:14.124435 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4772
I20260619 16:46:14.130487 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4906
I20260619 16:46:14.136561 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 4544
2026-06-19T16:46:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2375 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[5076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[5076](info): set up 2 sockets
Jun 19 16:46:14 dist-test-slave-cgwx krb5kdc[5076](info): commencing operation
krb5kdc: starting...
W20260619 16:46:15.210798 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:15 dist-test-slave-cgwx krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:15.332058 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:33875
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:33875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:15.440255 5089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:15.440519 5089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:15.440560 5089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:15.444572 5089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:15.444638 5089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:46:15.444655 5089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:15.444674 5089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:15.444689 5089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:15.444705 5089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:15.450076 5089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:33875
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:33875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:15.451144 5089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:15.452077 5089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:15.460610 5097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.460652 5095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.460652 5094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:15.460891 5089 server_base.cc:1034] running on GCE node
I20260619 16:46:15.461544 5089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:15.462754 5089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:15.463917 5089 hybrid_clock.cc:648] HybridClock initialized: now 1781887575463888 us; error 48 us; skew 500 ppm
Jun 19 16:46:15 dist-test-slave-cgwx krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:15.466706 5089 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:15.467727 5089 webserver.cc:460] Webserver started at http://127.3.233.190:39831/ using document root <none> and password file <none>
I20260619 16:46:15.468303 5089 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:15.468394 5089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:15.468623 5089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:15.470768 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "95ff4d5b5ba64b8d838d0fe726d17057"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "8ae9849b78d4f4e15349d005f17c39a7"
server_key_iv: "adf7ca9357a17b6836791332bfe5e7bc"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.471426 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "95ff4d5b5ba64b8d838d0fe726d17057"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "8ae9849b78d4f4e15349d005f17c39a7"
server_key_iv: "adf7ca9357a17b6836791332bfe5e7bc"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.475476 5089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:15.478013 5104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:15.479413 5089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:15.479555 5089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "95ff4d5b5ba64b8d838d0fe726d17057"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "8ae9849b78d4f4e15349d005f17c39a7"
server_key_iv: "adf7ca9357a17b6836791332bfe5e7bc"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.479672 5089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:15.501909 5089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:15.505016 5089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:15.505237 5089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:15.513368 5089 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:33875
I20260619 16:46:15.513398 5166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:33875 every 8 connection(s)
I20260619 16:46:15.514334 5089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:15.517369 5167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:15.518391 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 5089
I20260619 16:46:15.518528 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:15.518823 4006 external_mini_cluster.cc:1428] Setting key a0c3aeb152fedecb7963fa2fdb56138d
I20260619 16:46:15.524446 5167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057: Bootstrap starting.
I20260619 16:46:15.527405 5167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:15.528235 5167 log.cc:824] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:46:15 dist-test-slave-cgwx krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:15.530699 5167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057: No bootstrap required, opened a new log
I20260619 16:46:15.534639 5167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33875 } }
I20260619 16:46:15.534881 5167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:15.534993 5167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ff4d5b5ba64b8d838d0fe726d17057, State: Initialized, Role: FOLLOWER
I20260619 16:46:15.535432 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.520643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.3.233.190:33875)
0619 16:46:15.521097 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.521112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.521147 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:46:15.521784 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.521801 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.522137 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.522511 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.523480 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.525136 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.525994 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.526746 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.526944 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.529825 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.529831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.529845 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.529871 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.532248 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.532837 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.532842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.532847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.532963 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.533270 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.533273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.533274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.533673 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.533830 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.534150 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.534467 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260619 16:46:15.535475 5167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33875 } }
I20260619 16:46:15.535651 5167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:15.535718 5167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:15.535869 5167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:15.537037 5167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33875 } }
I20260619 16:46:15.537400 5167 leader_election.cc:302] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95ff4d5b5ba64b8d838d0fe726d17057; no voters:
I20260619 16:46:15.537786 5167 leader_election.cc:288] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:15.537895 5172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:15.538144 5172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [term 1 LEADER]: Becoming Leader. State: Replica: 95ff4d5b5ba64b8d838d0fe726d17057, State: Running, Role: LEADER
I20260619 16:46:15.538480 5172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33875 } }
I20260619 16:46:15.539161 5167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:15.540221 5173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33875 } } }
I20260619 16:46:15.540345 5173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:15.540714 5181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:15.540724 5174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ff4d5b5ba64b8d838d0fe726d17057. Latest consensus state: current_term: 1 leader_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ff4d5b5ba64b8d838d0fe726d17057" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33875 } } }
I20260619 16:46:15.540860 5174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:15.543943 5181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:15.549947 5181 catalog_manager.cc:1347] Generated new cluster ID: 6bce8582b5f84f058ddc81474ece187a
I20260619 16:46:15.550030 5181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:15.555888 5181 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:15.556819 5181 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:15.562417 5181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057: Generated new TSK 0
I20260619 16:46:15.563429 5181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:15.600337 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:33875
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:15.719619 5193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:15.719931 5193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:15.720009 5193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:15.724097 5193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:15.724191 5193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:15.724263 5193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:15.724349 5193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:15.729756 5193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:33875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:15.730882 5193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:15.731949 5193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:15.741591 5201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.741688 5199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.741694 5198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:15.741951 5193 server_base.cc:1034] running on GCE node
I20260619 16:46:15.742499 5193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:15.743199 5193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:15.744397 5193 hybrid_clock.cc:648] HybridClock initialized: now 1781887575744374 us; error 38 us; skew 500 ppm
Jun 19 16:46:15 dist-test-slave-cgwx krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:15.747303 5193 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:15.748136 5193 webserver.cc:460] Webserver started at http://127.3.233.129:37245/ using document root <none> and password file <none>
I20260619 16:46:15.748656 5193 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:15.748713 5193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:15.748898 5193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:15.750855 5193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "645d980798884b11a7bd027e54d84f4e"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "41e085dd263f64b46ba7d945b6d16c24"
server_key_iv: "2e845be20f57a6e61372ed8e86c3e01c"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.751418 5193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "645d980798884b11a7bd027e54d84f4e"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "41e085dd263f64b46ba7d945b6d16c24"
server_key_iv: "2e845be20f57a6e61372ed8e86c3e01c"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.755293 5193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:15.758041 5208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:15.759399 5193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:15.759588 5193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "645d980798884b11a7bd027e54d84f4e"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "41e085dd263f64b46ba7d945b6d16c24"
server_key_iv: "2e845be20f57a6e61372ed8e86c3e01c"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.759696 5193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:15.770653 5193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:15.774282 5193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:15.774569 5193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:15.776212 5193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:15.776266 5193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:15.776331 5193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:15.776368 5193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:15 dist-test-slave-cgwx krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:15.788602 5193 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:38641
I20260619 16:46:15.788627 5323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:38641 every 8 connection(s)
I20260619 16:46:15.789889 5193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:15.791836 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.777196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:38477 (local address 127.3.233.190:33875)
0619 16:46:15.777321 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.777324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.778535 (+ 1211us) server_negotiation.cc:407] Connection header received
0619 16:46:15.779713 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.779740 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.779815 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.779981 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.781645 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.782194 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.783801 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.784142 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.784287 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.787684 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.787688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.787691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.787721 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.789445 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.790142 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.790146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.790147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.790231 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.790694 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.790700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.790704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.790992 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.791096 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.791572 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.791703 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:46:15.792624 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.777808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33875 (local address 127.3.233.129:38477)
0619 16:46:15.778324 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.778367 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.779469 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.780224 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.780235 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.780691 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.781482 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.781505 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.782332 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.782335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.783665 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.783675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.784334 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.784338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.784547 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.785327 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.785344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.787420 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.789581 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.789587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.789603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.789987 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.790368 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.790373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.790375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.790557 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.791091 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.791096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.791320 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.791474 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.791738 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":158,"threads_started":1}
I20260619 16:46:15.794008 5324 heartbeater.cc:347] Connected to a master server at 127.3.233.190:33875
I20260619 16:46:15.794154 5324 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:15.794714 5324 heartbeater.cc:511] Master 127.3.233.190:33875 requested a full tablet report, sending...
I20260619 16:46:15.796411 5119 ts_manager.cc:194] Registered new tserver with Master: 645d980798884b11a7bd027e54d84f4e (127.3.233.129:38641)
I20260619 16:46:15.797940 5119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:38477
I20260619 16:46:15.797996 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 5193
I20260619 16:46:15.798094 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:15.798348 4006 external_mini_cluster.cc:1428] Setting key 6bcaaff70c154e9e418df36f9cfb460e
I20260619 16:46:15.804714 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.796364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33875 (local address 127.3.233.129:48791)
0619 16:46:15.796754 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:15.796769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:15.796867 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:15.797114 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:15.797117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:15.797334 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:15.797458 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.797466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.798524 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.798529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.799436 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:15.799446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.800097 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.800100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:15.800187 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.800742 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:15.800756 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:15.801090 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:15.803095 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.803098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.803099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.803296 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.803599 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:15.803601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:15.803603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:15.803712 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:15.804218 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:15.804221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:15.804305 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 16:46:15.804408 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:46:15.804528 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":1,"thread_start_us":147,"threads_started":1}
I20260619 16:46:15.804852 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:15.796443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:48791 (local address 127.3.233.190:33875)
0619 16:46:15.796626 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:15.796631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:15.796841 (+ 210us) server_negotiation.cc:407] Connection header received
0619 16:46:15.796953 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:15.796955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:15.797002 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:15.797073 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:15.797595 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.798340 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.799570 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:15.799875 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:15.799979 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:15.801226 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:15.801229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:15.801232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:15.801254 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:15.802984 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.803395 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.803401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.803406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.803498 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:15.803828 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:15.803834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:15.803838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:15.804092 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:15.804201 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:15.804491 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:46:15.804622 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:15.825970 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:33875
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:15.936713 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:15.937007 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:15.937088 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:15.941637 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:15.941720 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:15.941789 5333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:15.941865 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:15.947124 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:33875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:15.948149 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:15.949183 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:15.959025 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.959049 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:15.959049 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:15.959339 5333 server_base.cc:1034] running on GCE node
I20260619 16:46:15.959738 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:15.960395 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:15.961560 5333 hybrid_clock.cc:648] HybridClock initialized: now 1781887575961545 us; error 31 us; skew 500 ppm
Jun 19 16:46:15 dist-test-slave-cgwx krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:15.964699 5333 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:15.965773 5333 webserver.cc:460] Webserver started at http://127.3.233.130:42343/ using document root <none> and password file <none>
I20260619 16:46:15.966387 5333 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:15.966454 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:15.966749 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:15.969084 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "6c66fc51eca543c5a71496269d53a8e9"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "85055b59f77e7c0d3ba9e946e6765fe7"
server_key_iv: "06699689dee9b81138ac64ed3c37810b"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.969713 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "6c66fc51eca543c5a71496269d53a8e9"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "85055b59f77e7c0d3ba9e946e6765fe7"
server_key_iv: "06699689dee9b81138ac64ed3c37810b"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.973934 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 16:46:15.976811 5348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:15.978341 5333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:15.978478 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "6c66fc51eca543c5a71496269d53a8e9"
format_stamp: "Formatted at 2026-06-19 16:46:15 on dist-test-slave-cgwx"
server_key: "85055b59f77e7c0d3ba9e946e6765fe7"
server_key_iv: "06699689dee9b81138ac64ed3c37810b"
server_key_version: "encryptionkey@0"
I20260619 16:46:15.978602 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:16.001343 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:16.005234 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:16.005508 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:16.007252 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:16.007309 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:16.007405 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:16.007445 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:16.019758 5333 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:39655
I20260619 16:46:16.019760 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:39655 every 8 connection(s)
I20260619 16:46:16.020864 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:16.022240 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.008146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:46847 (local address 127.3.233.190:33875)
0619 16:46:16.008294 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.008299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.009344 (+ 1045us) server_negotiation.cc:407] Connection header received
0619 16:46:16.010759 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.010765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.010827 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.010928 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.012290 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.012827 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.014010 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.014233 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.014343 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.017813 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.017819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.017822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.017853 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.019726 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.020267 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.020271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.020273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.020369 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.020823 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.020830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.020835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.021095 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.021216 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.021936 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.022072 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:46:16.022495 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 5333
I20260619 16:46:16.022624 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:16.022910 4006 external_mini_cluster.cc:1428] Setting key af2f7173dd5456271183c36ccc5c75cd
I20260619 16:46:16.023175 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.008592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33875 (local address 127.3.233.130:46847)
0619 16:46:16.009143 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.009210 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.010473 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.011147 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.011155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.011509 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:16.012121 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.012142 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.012962 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.012965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.013878 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.013887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.014397 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.014401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.014560 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.015252 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:16.015290 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:16.017640 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:16.019868 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.019873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.019887 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.020156 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.020553 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.020556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.020557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.020707 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.021250 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:16.021259 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:16.021645 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.021875 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.022137 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260619 16:46:16.024513 5464 heartbeater.cc:347] Connected to a master server at 127.3.233.190:33875
I20260619 16:46:16.024647 5464 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:16.025204 5464 heartbeater.cc:511] Master 127.3.233.190:33875 requested a full tablet report, sending...
I20260619 16:46:16.026486 5119 ts_manager.cc:194] Registered new tserver with Master: 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655)
I20260619 16:46:16.027415 5119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:46847
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:46:16.035352 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.026656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33875 (local address 127.3.233.130:53083)
0619 16:46:16.027015 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.027029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.027142 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.027364 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.027367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.027570 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:16.027686 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.027696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.028689 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.028696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.029845 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.029855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.030591 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.030594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.030667 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.031257 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:16.031269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:16.031652 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:16.033624 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.033628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.033629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.033937 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.034247 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.034250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.034251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.034334 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.034852 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:16.034855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:16.034998 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.035099 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.035206 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":88,"threads_started":1}
I20260619 16:46:16.035549 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.026695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:53083 (local address 127.3.233.190:33875)
0619 16:46:16.026806 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.026811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.027104 (+ 293us) server_negotiation.cc:407] Connection header received
0619 16:46:16.027220 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.027223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.027267 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.027350 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.027818 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.028540 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.029969 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.030415 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.030546 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.031803 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.031807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.031810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.031837 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.033500 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.034046 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.034052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.034056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.034143 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.034411 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.034413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.034414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.034698 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.034806 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.035248 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.035345 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:16.052032 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:33875
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:16.162032 5473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:16.162326 5473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:16.162423 5473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:16.166937 5473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:16.167003 5473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:16.167083 5473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:16.167145 5473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:16.172513 5473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:33875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:16.173568 5473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:16.174557 5473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:16.184881 5478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:16.184895 5481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:16.184998 5479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:16.185092 5473 server_base.cc:1034] running on GCE node
I20260619 16:46:16.185562 5473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:16.186241 5473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:16.187426 5473 hybrid_clock.cc:648] HybridClock initialized: now 1781887576187404 us; error 31 us; skew 500 ppm
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:16.190274 5473 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:16.191257 5473 webserver.cc:460] Webserver started at http://127.3.233.131:37895/ using document root <none> and password file <none>
I20260619 16:46:16.191815 5473 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:16.191907 5473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:16.192137 5473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:16.194227 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "c2606ae1e43b4030b8306a10e940c976"
format_stamp: "Formatted at 2026-06-19 16:46:16 on dist-test-slave-cgwx"
server_key: "700992bbd54d02b62307d26285e4ae35"
server_key_iv: "d83f5f095f042068a840ed7183baf8ed"
server_key_version: "encryptionkey@0"
I20260619 16:46:16.194787 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "c2606ae1e43b4030b8306a10e940c976"
format_stamp: "Formatted at 2026-06-19 16:46:16 on dist-test-slave-cgwx"
server_key: "700992bbd54d02b62307d26285e4ae35"
server_key_iv: "d83f5f095f042068a840ed7183baf8ed"
server_key_version: "encryptionkey@0"
I20260619 16:46:16.198603 5473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:46:16.201128 5488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:16.202435 5473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:16.202596 5473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "c2606ae1e43b4030b8306a10e940c976"
format_stamp: "Formatted at 2026-06-19 16:46:16 on dist-test-slave-cgwx"
server_key: "700992bbd54d02b62307d26285e4ae35"
server_key_iv: "d83f5f095f042068a840ed7183baf8ed"
server_key_version: "encryptionkey@0"
I20260619 16:46:16.202716 5473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:16.217007 5473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:16.220528 5473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:16.220741 5473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:16.222270 5473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:16.222352 5473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:16.222437 5473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:16.222476 5473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887576, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:16.233426 5473 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:44911
I20260619 16:46:16.234283 5473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:16.234826 5603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:44911 every 8 connection(s)
I20260619 16:46:16.235944 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.223282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:44605 (local address 127.3.233.190:33875)
0619 16:46:16.223486 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.223490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.224433 (+ 943us) server_negotiation.cc:407] Connection header received
0619 16:46:16.225675 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.225678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.225737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.225830 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.227511 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.228202 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.229475 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.229710 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.229819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.232358 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.232361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.232363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.232385 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.233921 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.234417 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.234422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.234429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.234506 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.234902 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.234907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.234910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.235148 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.235243 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.235730 (+ 487us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.235827 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:46:16.236828 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.223710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33875 (local address 127.3.233.131:44605)
0619 16:46:16.224230 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.224317 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.225390 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.226062 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.226074 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.226559 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:16.227331 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.227354 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.228334 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.228339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.229356 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.229366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.229867 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.229871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.230032 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.230649 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:16.230664 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:16.232178 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:16.234038 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.234044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.234058 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.234311 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.234603 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.234606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.234608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.234780 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.235260 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:16.235265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:16.235518 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.235666 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.235917 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":145,"threads_started":1}
I20260619 16:46:16.238085 5604 heartbeater.cc:347] Connected to a master server at 127.3.233.190:33875
I20260619 16:46:16.238209 5604 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:16.238373 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 5473
I20260619 16:46:16.238512 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:16.238834 4006 external_mini_cluster.cc:1428] Setting key 5a23b891ff67289c092df848afce841f
I20260619 16:46:16.238834 5604 heartbeater.cc:511] Master 127.3.233.190:33875 requested a full tablet report, sending...
I20260619 16:46:16.239945 5120 ts_manager.cc:194] Registered new tserver with Master: c2606ae1e43b4030b8306a10e940c976 (127.3.233.131:44911)
I20260619 16:46:16.240590 5120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:44605
I20260619 16:46:16.240947 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:16.249398 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.241127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:37911 (local address 127.3.233.190:33875)
0619 16:46:16.241250 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.241254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.241510 (+ 256us) server_negotiation.cc:407] Connection header received
0619 16:46:16.241594 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.241599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.241647 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.241746 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.242109 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.242958 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.244251 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.244454 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.244534 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.245875 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.245878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.245880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.245900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.247794 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.248312 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.248314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.248316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.248384 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.248673 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.248675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.248676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.248829 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.248901 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.249201 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.249287 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:46:16.249657 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.241153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33875 (local address 127.3.233.131:37911)
0619 16:46:16.241420 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.241437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.241520 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.241748 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.241750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.241913 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:16.242009 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.242014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.243080 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.243086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.244118 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.244132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.244573 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.244579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.244714 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.245227 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:16.245246 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:16.245735 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:16.247926 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.247931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.247936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.248215 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.248473 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.248479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.248481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.248584 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.248919 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:16.248922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:16.249037 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.249156 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.249336 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:16.269496 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.260606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.3.233.190:33875)
0619 16:46:16.260738 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.260742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.260756 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:16.260793 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.260795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.260831 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.260923 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.261696 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.262225 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.263238 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.263491 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.263587 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.266237 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.266240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.266242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.266265 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.267775 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.268266 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.268269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.268270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.268350 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.268663 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.268666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.268667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.268843 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.268966 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.269260 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.269361 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:46:16.272895 5120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:16.276001 5120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:16.290971 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.285075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.3.233.131:44911)
0619 16:46:16.285638 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.285656 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.285971 (+ 315us) server_negotiation.cc:407] Connection header received
0619 16:46:16.286058 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.286063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.286274 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.286519 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:16.287111 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.288285 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.289655 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.290244 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.290374 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.290541 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.290676 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.290776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":126,"threads_started":1}
I20260619 16:46:16.290992 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.285119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39655 (local address 127.0.0.1:44256)
0619 16:46:16.285939 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.285953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.286054 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.286862 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.286865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.286879 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:16.286985 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.286992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.288020 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.288025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.289330 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.289339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.290436 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.290440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.290541 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.290567 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.290618 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.290776 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260619 16:46:16.291182 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.285665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:38641 (local address 127.0.0.1:56400)
0619 16:46:16.286395 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.286408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.286495 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.287105 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.287107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.287125 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:16.287234 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.287241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.288574 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.288580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.289664 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.289674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.290737 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.290740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.290826 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.290847 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.290950 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.291029 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":83,"threads_started":1}
I20260619 16:46:16.291246 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.285848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.3.233.129:38641)
0619 16:46:16.286240 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.286256 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.286588 (+ 332us) server_negotiation.cc:407] Connection header received
0619 16:46:16.286656 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.286661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.286808 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.286953 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:16.287365 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.288465 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.289783 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.290564 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.290662 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.290815 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.290996 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.291106 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":120,"threads_started":1}
I20260619 16:46:16.291615 5621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.284895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:44911 (local address 127.0.0.1:50352)
0619 16:46:16.285554 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.285625 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.285808 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.286737 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.286742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.286766 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:16.286915 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.286971 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.288403 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.288406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.289559 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.289567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.290373 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.290376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.290478 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.290515 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.290646 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.291504 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":813,"thread_start_us":107,"threads_started":1}
I20260619 16:46:16.291867 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.286043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.3.233.130:39655)
0619 16:46:16.286441 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.286453 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.286487 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:46:16.286560 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.286566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.286742 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.286894 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:16.287098 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.287901 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.289462 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.290300 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.291460 (+ 1160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.291595 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.291633 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.291735 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":92,"threads_started":1}
I20260619 16:46:16.294219 5398 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 (DEFAULT_TABLE table=test-table [id=3665cdf6567645c1b43eba79b63577c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:16.294247 5534 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 (DEFAULT_TABLE table=test-table [id=3665cdf6567645c1b43eba79b63577c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:16.294232 5258 tablet_service.cc:1467] Processing CreateTablet for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 (DEFAULT_TABLE table=test-table [id=3665cdf6567645c1b43eba79b63577c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:16.295305 5534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.295302 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.295302 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.301599 5627 tablet_bootstrap.cc:492] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Bootstrap starting.
I20260619 16:46:16.303092 5628 tablet_bootstrap.cc:492] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Bootstrap starting.
I20260619 16:46:16.303247 5629 tablet_bootstrap.cc:492] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Bootstrap starting.
I20260619 16:46:16.304169 5627 tablet_bootstrap.cc:654] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.305172 5627 log.cc:824] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:16.305405 5629 tablet_bootstrap.cc:654] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.305524 5628 tablet_bootstrap.cc:654] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.306259 5629 log.cc:824] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:16.306388 5628 log.cc:824] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:16.307497 5627 tablet_bootstrap.cc:492] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: No bootstrap required, opened a new log
I20260619 16:46:16.307708 5627 ts_tablet_manager.cc:1397] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:46:16.308274 5628 tablet_bootstrap.cc:492] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: No bootstrap required, opened a new log
I20260619 16:46:16.308290 5629 tablet_bootstrap.cc:492] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: No bootstrap required, opened a new log
I20260619 16:46:16.308461 5629 ts_tablet_manager.cc:1397] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:46:16.308485 5628 ts_tablet_manager.cc:1397] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:46:16.311820 5628 raft_consensus.cc:348] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.311828 5627 raft_consensus.cc:348] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.312078 5627 raft_consensus.cc:374] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.312078 5628 raft_consensus.cc:374] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.312140 5628 raft_consensus.cc:729] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645d980798884b11a7bd027e54d84f4e, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.312170 5627 raft_consensus.cc:729] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c66fc51eca543c5a71496269d53a8e9, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.312306 5629 raft_consensus.cc:348] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.312541 5629 raft_consensus.cc:374] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.312625 5629 raft_consensus.cc:729] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2606ae1e43b4030b8306a10e940c976, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.312666 5627 consensus_queue.cc:260] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.312654 5628 consensus_queue.cc:260] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.313060 5629 consensus_queue.cc:260] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.313290 5324 heartbeater.cc:503] Master 127.3.233.190:33875 was elected leader, sending a full tablet report...
I20260619 16:46:16.313686 5604 heartbeater.cc:503] Master 127.3.233.190:33875 was elected leader, sending a full tablet report...
I20260619 16:46:16.313709 5628 ts_tablet_manager.cc:1428] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:46:16.314236 5627 ts_tablet_manager.cc:1428] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:46:16.314594 5464 heartbeater.cc:503] Master 127.3.233.190:33875 was elected leader, sending a full tablet report...
I20260619 16:46:16.314857 5629 ts_tablet_manager.cc:1428] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:46:16.358461 5635 raft_consensus.cc:482] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:16.358776 5635 raft_consensus.cc:504] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.360047 5635 leader_election.cc:288] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655), 645d980798884b11a7bd027e54d84f4e (127.3.233.129:38641)
I20260619 16:46:16.364871 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.360557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:40057 (local address 127.3.233.130:39655)
0619 16:46:16.360713 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.360718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.360783 (+ 65us) server_negotiation.cc:407] Connection header received
0619 16:46:16.361008 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.361012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.361070 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.361158 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:16.361546 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.362145 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.363600 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.364247 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.364361 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.364411 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.364672 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.364743 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:46:16.364871 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.360397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39655 (local address 127.3.233.131:40057)
0619 16:46:16.360696 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.360713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.360829 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.361229 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.361232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.361257 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:16.361402 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.361416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.362268 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.362272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.363454 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.363463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.364376 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.364379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.364523 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.364555 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.364654 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.364736 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260619 16:46:16.365346 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.360710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:38641 (local address 127.3.233.131:39199)
0619 16:46:16.361069 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.361082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.361174 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.361423 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.361426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.361443 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:16.361598 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.361607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.362528 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.362530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.363991 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.364002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.364945 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.364949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.365037 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.365058 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.365132 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.365189 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260619 16:46:16.365363 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.360857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:39199 (local address 127.3.233.129:38641)
0619 16:46:16.361018 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.361022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.361185 (+ 163us) server_negotiation.cc:407] Connection header received
0619 16:46:16.361251 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.361254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.361300 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.361386 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:16.361730 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.362331 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.364153 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.364703 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.364794 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.364837 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.365172 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.365248 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:46:16.365931 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffd27f1b11a4da6af67ae2c8d0a6415" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c66fc51eca543c5a71496269d53a8e9" is_pre_election: true
I20260619 16:46:16.366081 5277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffd27f1b11a4da6af67ae2c8d0a6415" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645d980798884b11a7bd027e54d84f4e" is_pre_election: true
I20260619 16:46:16.366256 5418 raft_consensus.cc:2454] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 0.
I20260619 16:46:16.366417 5277 raft_consensus.cc:2454] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 0.
I20260619 16:46:16.366771 5490 leader_election.cc:302] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c66fc51eca543c5a71496269d53a8e9, c2606ae1e43b4030b8306a10e940c976; no voters:
I20260619 16:46:16.367071 5635 raft_consensus.cc:2781] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:16.367166 5635 raft_consensus.cc:482] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:16.367223 5635 raft_consensus.cc:3037] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.368251 5635 raft_consensus.cc:504] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.368595 5635 leader_election.cc:288] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 election: Requested vote from peers 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655), 645d980798884b11a7bd027e54d84f4e (127.3.233.129:38641)
I20260619 16:46:16.368927 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffd27f1b11a4da6af67ae2c8d0a6415" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c66fc51eca543c5a71496269d53a8e9"
I20260619 16:46:16.369035 5277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ffd27f1b11a4da6af67ae2c8d0a6415" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645d980798884b11a7bd027e54d84f4e"
I20260619 16:46:16.369028 5418 raft_consensus.cc:3037] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.369174 5277 raft_consensus.cc:3037] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.370162 5277 raft_consensus.cc:2454] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 1.
I20260619 16:46:16.370529 5490 leader_election.cc:302] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645d980798884b11a7bd027e54d84f4e, c2606ae1e43b4030b8306a10e940c976; no voters:
I20260619 16:46:16.370657 5418 raft_consensus.cc:2454] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 1.
I20260619 16:46:16.370716 5635 raft_consensus.cc:2781] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:16.370981 5635 raft_consensus.cc:686] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 1 LEADER]: Becoming Leader. State: Replica: c2606ae1e43b4030b8306a10e940c976, State: Running, Role: LEADER
I20260619 16:46:16.371305 5635 consensus_queue.cc:237] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } }
I20260619 16:46:16.374604 5120 catalog_manager.cc:5495] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2606ae1e43b4030b8306a10e940c976 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "c2606ae1e43b4030b8306a10e940c976" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:16.398058 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.393836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.3.233.131:44911)
0619 16:46:16.393969 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.393972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.394007 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:46:16.394228 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.394231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.394285 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.394371 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:16.394814 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.395419 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.396350 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.396636 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.396717 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.397116 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:16.397621 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:16.397724 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.397870 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.397937 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:46:16.404053 5418 raft_consensus.cc:1261] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Refusing update from remote peer c2606ae1e43b4030b8306a10e940c976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:16.404053 5277 raft_consensus.cc:1261] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Refusing update from remote peer c2606ae1e43b4030b8306a10e940c976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:16.404893 5637 consensus_queue.cc:1035] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:16.405085 5635 consensus_queue.cc:1035] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:16.414074 5641 mvcc.cc:205] Tried to move back new op lower bound from 7298611512943046656 to 7298611512820006912. Current Snapshot: MvccSnapshot[applied={T|T < 7298611512943046656}]
I20260619 16:46:16.416431 5642 mvcc.cc:205] Tried to move back new op lower bound from 7298611512943046656 to 7298611512820006912. Current Snapshot: MvccSnapshot[applied={T|T < 7298611512943046656 or (T in {7298611512943046656})}]
I20260619 16:46:16.424919 5121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:16.425087 5121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:16.427762 5121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057: Sending DeleteTablet for 3 replicas of tablet 4ffd27f1b11a4da6af67ae2c8d0a6415
I20260619 16:46:16.428563 5534 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50352
I20260619 16:46:16.428714 5398 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:44256
I20260619 16:46:16.428759 5258 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56400
I20260619 16:46:16.429203 5651 tablet_replica.cc:330] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: stopping tablet replica
I20260619 16:46:16.429317 5652 tablet_replica.cc:330] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: stopping tablet replica
I20260619 16:46:16.429586 5651 raft_consensus.cc:2229] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:16.429679 5652 raft_consensus.cc:2229] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:16.429968 5651 raft_consensus.cc:2258] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:16.430073 5652 raft_consensus.cc:2258] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:16.430105 5120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:16.430547 5120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:16.431677 5651 ts_tablet_manager.cc:1905] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:16.431677 5652 ts_tablet_manager.cc:1905] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:16.434391 5651 ts_tablet_manager.cc:1918] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:16.434554 5651 log.cc:1196] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/4ffd27f1b11a4da6af67ae2c8d0a6415
I20260619 16:46:16.434584 5652 ts_tablet_manager.cc:1918] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:16.434809 5652 log.cc:1196] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/4ffd27f1b11a4da6af67ae2c8d0a6415
I20260619 16:46:16.434943 5651 ts_tablet_manager.cc:1939] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 6c66fc51eca543c5a71496269d53a8e9: Deleting consensus metadata
I20260619 16:46:16.435359 5652 ts_tablet_manager.cc:1939] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P 645d980798884b11a7bd027e54d84f4e: Deleting consensus metadata
I20260619 16:46:16.436228 5106 catalog_manager.cc:4841] TS 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655): tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 (table test-table [id=3665cdf6567645c1b43eba79b63577c9]) successfully deleted
I20260619 16:46:16.436553 5106 catalog_manager.cc:4841] TS 645d980798884b11a7bd027e54d84f4e (127.3.233.129:38641): tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 (table test-table [id=3665cdf6567645c1b43eba79b63577c9]) successfully deleted
I20260619 16:46:16.437213 5398 tablet_service.cc:1467] Processing CreateTablet for tablet f36b9b6d4619437e8f815b444ab9141d (DEFAULT_TABLE table=test-table [id=64689f59a7f34a71893530426808cbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:16.437208 5258 tablet_service.cc:1467] Processing CreateTablet for tablet f36b9b6d4619437e8f815b444ab9141d (DEFAULT_TABLE table=test-table [id=64689f59a7f34a71893530426808cbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:16.437448 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36b9b6d4619437e8f815b444ab9141d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.437455 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36b9b6d4619437e8f815b444ab9141d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.437482 5534 tablet_service.cc:1467] Processing CreateTablet for tablet f36b9b6d4619437e8f815b444ab9141d (DEFAULT_TABLE table=test-table [id=64689f59a7f34a71893530426808cbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:16.437785 5534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f36b9b6d4619437e8f815b444ab9141d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.439767 5627 tablet_bootstrap.cc:492] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Bootstrap starting.
I20260619 16:46:16.439889 5628 tablet_bootstrap.cc:492] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Bootstrap starting.
I20260619 16:46:16.440730 5627 tablet_bootstrap.cc:654] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.440953 5628 tablet_bootstrap.cc:654] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.441167 5629 tablet_bootstrap.cc:492] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Bootstrap starting.
I20260619 16:46:16.442164 5629 tablet_bootstrap.cc:654] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.442646 5627 tablet_bootstrap.cc:492] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: No bootstrap required, opened a new log
I20260619 16:46:16.442745 5627 ts_tablet_manager.cc:1397] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:16.443312 5627 raft_consensus.cc:348] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.443440 5627 raft_consensus.cc:374] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.443478 5627 raft_consensus.cc:729] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c66fc51eca543c5a71496269d53a8e9, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.443552 5629 tablet_bootstrap.cc:492] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: No bootstrap required, opened a new log
I20260619 16:46:16.443631 5629 ts_tablet_manager.cc:1397] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:16.443615 5627 consensus_queue.cc:260] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.443907 5627 ts_tablet_manager.cc:1428] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:16.444044 5629 raft_consensus.cc:348] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.444165 5629 raft_consensus.cc:374] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.444195 5629 raft_consensus.cc:729] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2606ae1e43b4030b8306a10e940c976, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.444300 5629 consensus_queue.cc:260] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.444384 5650 tablet_replica.cc:330] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: stopping tablet replica
I20260619 16:46:16.445117 5650 raft_consensus.cc:2229] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:16.446084 5629 ts_tablet_manager.cc:1428] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:16.447116 5650 raft_consensus.cc:2258] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:16.447818 5628 tablet_bootstrap.cc:492] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: No bootstrap required, opened a new log
I20260619 16:46:16.447912 5628 ts_tablet_manager.cc:1397] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 16:46:16.448532 5628 raft_consensus.cc:348] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.448645 5628 raft_consensus.cc:374] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.448678 5628 raft_consensus.cc:729] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645d980798884b11a7bd027e54d84f4e, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.448866 5650 ts_tablet_manager.cc:1905] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:16.448913 5628 consensus_queue.cc:260] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.449245 5628 ts_tablet_manager.cc:1428] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:16.451912 5650 ts_tablet_manager.cc:1918] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:16.452019 5650 log.cc:1196] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/4ffd27f1b11a4da6af67ae2c8d0a6415
I20260619 16:46:16.452394 5650 ts_tablet_manager.cc:1939] T 4ffd27f1b11a4da6af67ae2c8d0a6415 P c2606ae1e43b4030b8306a10e940c976: Deleting consensus metadata
I20260619 16:46:16.453500 5108 catalog_manager.cc:4841] TS c2606ae1e43b4030b8306a10e940c976 (127.3.233.131:44911): tablet 4ffd27f1b11a4da6af67ae2c8d0a6415 (table test-table [id=3665cdf6567645c1b43eba79b63577c9]) successfully deleted
I20260619 16:46:16.482178 5645 raft_consensus.cc:482] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:16.482357 5645 raft_consensus.cc:504] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.482966 5645 leader_election.cc:288] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 645d980798884b11a7bd027e54d84f4e (127.3.233.129:38641), 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655)
I20260619 16:46:16.483397 5277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36b9b6d4619437e8f815b444ab9141d" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645d980798884b11a7bd027e54d84f4e" is_pre_election: true
I20260619 16:46:16.483419 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36b9b6d4619437e8f815b444ab9141d" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c66fc51eca543c5a71496269d53a8e9" is_pre_election: true
I20260619 16:46:16.483551 5277 raft_consensus.cc:2454] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 0.
I20260619 16:46:16.483597 5418 raft_consensus.cc:2454] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 0.
I20260619 16:46:16.483928 5490 leader_election.cc:302] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c66fc51eca543c5a71496269d53a8e9, c2606ae1e43b4030b8306a10e940c976; no voters:
I20260619 16:46:16.484151 5645 raft_consensus.cc:2781] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:16.484230 5645 raft_consensus.cc:482] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:16.484287 5645 raft_consensus.cc:3037] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Advancing to term 1
W20260619 16:46:16.485428 5605 tablet.cc:2369] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:16.485499 5645 raft_consensus.cc:504] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.485837 5645 leader_election.cc:288] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 election: Requested vote from peers 645d980798884b11a7bd027e54d84f4e (127.3.233.129:38641), 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655)
I20260619 16:46:16.486254 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36b9b6d4619437e8f815b444ab9141d" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c66fc51eca543c5a71496269d53a8e9"
I20260619 16:46:16.486254 5277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f36b9b6d4619437e8f815b444ab9141d" candidate_uuid: "c2606ae1e43b4030b8306a10e940c976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645d980798884b11a7bd027e54d84f4e"
I20260619 16:46:16.486385 5418 raft_consensus.cc:3037] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.486385 5277 raft_consensus.cc:3037] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.487659 5418 raft_consensus.cc:2454] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 1.
I20260619 16:46:16.487746 5277 raft_consensus.cc:2454] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2606ae1e43b4030b8306a10e940c976 in term 1.
I20260619 16:46:16.488013 5490 leader_election.cc:302] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c66fc51eca543c5a71496269d53a8e9, c2606ae1e43b4030b8306a10e940c976; no voters:
I20260619 16:46:16.488200 5645 raft_consensus.cc:2781] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:16.488327 5645 raft_consensus.cc:686] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 1 LEADER]: Becoming Leader. State: Replica: c2606ae1e43b4030b8306a10e940c976, State: Running, Role: LEADER
I20260619 16:46:16.488466 5645 consensus_queue.cc:237] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.490365 5120 catalog_manager.cc:5495] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2606ae1e43b4030b8306a10e940c976 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "c2606ae1e43b4030b8306a10e940c976" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } health_report { overall_health: HEALTHY } } }
W20260619 16:46:16.522140 5465 tablet.cc:2369] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887575, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:16.527194 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.516858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33875 (local address 127.0.0.1:47786)
0619 16:46:16.517235 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.517249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.517374 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.517660 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.517663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.518180 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:16.518297 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.518304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.519436 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.519439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.520272 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.520283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.520862 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.520866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.520950 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.521629 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:16.521644 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:16.523420 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:16.525521 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.525529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.525534 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.525816 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.526113 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:16.526116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:16.526119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:16.526211 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:16.526607 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:16.526613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:16.526730 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.526842 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.527005 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":107,"threads_started":1}
I20260619 16:46:16.527289 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.516943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.3.233.190:33875)
0619 16:46:16.517074 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.517079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.517346 (+ 267us) server_negotiation.cc:407] Connection header received
0619 16:46:16.517477 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.517480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.517526 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.517612 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.518438 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.519308 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.520392 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.520737 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.520832 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.523630 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.523634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.523637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.523662 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.525410 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.525917 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.525921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.525923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.526005 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.526297 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.526300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.526302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.526508 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.526603 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.526943 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.527078 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:46:16.532558 5120 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:47786:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:46:16.533455 5120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:16.538831 5258 tablet_service.cc:1467] Processing CreateTablet for tablet fe349d9cecf94f158c6ef84325e8fc1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a922703ae65b457a96d80b2929d900db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:16.539172 5534 tablet_service.cc:1467] Processing CreateTablet for tablet fe349d9cecf94f158c6ef84325e8fc1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a922703ae65b457a96d80b2929d900db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:16.539358 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe349d9cecf94f158c6ef84325e8fc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.539235 5398 tablet_service.cc:1467] Processing CreateTablet for tablet fe349d9cecf94f158c6ef84325e8fc1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a922703ae65b457a96d80b2929d900db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:16.539486 5534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe349d9cecf94f158c6ef84325e8fc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.539619 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe349d9cecf94f158c6ef84325e8fc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:16.542214 5629 tablet_bootstrap.cc:492] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: Bootstrap starting.
I20260619 16:46:16.542562 5628 tablet_bootstrap.cc:492] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: Bootstrap starting.
W20260619 16:46:16.542794 5325 tablet.cc:2369] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:16.543277 5629 tablet_bootstrap.cc:654] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.543605 5628 tablet_bootstrap.cc:654] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.544238 5627 tablet_bootstrap.cc:492] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: Bootstrap starting.
I20260619 16:46:16.544777 5629 tablet_bootstrap.cc:492] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: No bootstrap required, opened a new log
I20260619 16:46:16.544857 5629 ts_tablet_manager.cc:1397] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:16.545221 5627 tablet_bootstrap.cc:654] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:16.545311 5629 raft_consensus.cc:348] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.545432 5629 raft_consensus.cc:374] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.545462 5629 raft_consensus.cc:729] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2606ae1e43b4030b8306a10e940c976, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.545616 5629 consensus_queue.cc:260] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.545882 5629 ts_tablet_manager.cc:1428] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:16.546057 5645 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.546178 5645 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:16.546811 5628 tablet_bootstrap.cc:492] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: No bootstrap required, opened a new log
I20260619 16:46:16.546895 5628 ts_tablet_manager.cc:1397] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:46:16.547418 5627 tablet_bootstrap.cc:492] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: No bootstrap required, opened a new log
I20260619 16:46:16.547399 5628 raft_consensus.cc:348] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.547502 5628 raft_consensus.cc:374] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.547509 5627 ts_tablet_manager.cc:1397] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:16.547539 5628 raft_consensus.cc:729] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645d980798884b11a7bd027e54d84f4e, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.547665 5628 consensus_queue.cc:260] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.547901 5628 ts_tablet_manager.cc:1428] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:16.547936 5627 raft_consensus.cc:348] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.548038 5627 raft_consensus.cc:374] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:16.548074 5627 raft_consensus.cc:729] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c66fc51eca543c5a71496269d53a8e9, State: Initialized, Role: FOLLOWER
I20260619 16:46:16.548149 5633 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.548211 5627 consensus_queue.cc:260] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.548272 5633 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:16.548462 5634 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.548600 5634 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:16.548476 5627 ts_tablet_manager.cc:1428] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:16.606755 5633 raft_consensus.cc:482] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:16.606992 5633 raft_consensus.cc:504] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.608498 5633 leader_election.cc:288] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655), c2606ae1e43b4030b8306a10e940c976 (127.3.233.131:44911)
I20260619 16:46:16.614394 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.608855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:45719 (local address 127.3.233.130:39655)
0619 16:46:16.609004 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.609009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.609285 (+ 276us) server_negotiation.cc:407] Connection header received
0619 16:46:16.609447 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.609450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.609592 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.609668 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:16.610046 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.610847 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.612780 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.613562 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.613691 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.613744 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.614161 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.614250 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:46:16.614394 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.608909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:46769 (local address 127.3.233.131:44911)
0619 16:46:16.609068 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.609072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.609366 (+ 294us) server_negotiation.cc:407] Connection header received
0619 16:46:16.609462 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.609465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.609513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.609602 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:16.610046 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.610847 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.612780 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.613506 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.613676 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.613739 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.614161 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.614250 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:46:16.614439 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:44911 (local address 127.3.233.129:46769)
0619 16:46:16.609269 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.609283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.609375 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.609624 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.609626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.609668 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:16.609881 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.609889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.611015 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.611019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.612595 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.612608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.613712 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.613716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.613946 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.614022 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.614117 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.614234 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":267,"spinlock_wait_cycles":20864,"thread_start_us":118,"threads_started":1}
I20260619 16:46:16.614439 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.608693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39655 (local address 127.3.233.129:45719)
0619 16:46:16.609169 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.609186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.609324 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.609697 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.609701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.609738 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:16.609881 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.609889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.611015 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.611019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.612595 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.612608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.613712 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.613716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.613985 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.614021 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.614110 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.614234 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":220,"spinlock_wait_cycles":3840,"thread_start_us":129,"threads_started":1}
I20260619 16:46:16.615017 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe349d9cecf94f158c6ef84325e8fc1f" candidate_uuid: "645d980798884b11a7bd027e54d84f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c66fc51eca543c5a71496269d53a8e9" is_pre_election: true
I20260619 16:46:16.615087 5552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe349d9cecf94f158c6ef84325e8fc1f" candidate_uuid: "645d980798884b11a7bd027e54d84f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2606ae1e43b4030b8306a10e940c976" is_pre_election: true
I20260619 16:46:16.615204 5418 raft_consensus.cc:2454] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645d980798884b11a7bd027e54d84f4e in term 0.
I20260619 16:46:16.615397 5552 raft_consensus.cc:2454] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645d980798884b11a7bd027e54d84f4e in term 0.
I20260619 16:46:16.615811 5210 leader_election.cc:302] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 645d980798884b11a7bd027e54d84f4e, 6c66fc51eca543c5a71496269d53a8e9, c2606ae1e43b4030b8306a10e940c976; no voters:
I20260619 16:46:16.616106 5633 raft_consensus.cc:2781] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:16.616169 5633 raft_consensus.cc:482] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:16.616209 5633 raft_consensus.cc:3037] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.617228 5633 raft_consensus.cc:504] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.617614 5633 leader_election.cc:288] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [CANDIDATE]: Term 1 election: Requested vote from peers 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655), c2606ae1e43b4030b8306a10e940c976 (127.3.233.131:44911)
I20260619 16:46:16.617991 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe349d9cecf94f158c6ef84325e8fc1f" candidate_uuid: "645d980798884b11a7bd027e54d84f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c66fc51eca543c5a71496269d53a8e9"
I20260619 16:46:16.618055 5552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe349d9cecf94f158c6ef84325e8fc1f" candidate_uuid: "645d980798884b11a7bd027e54d84f4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2606ae1e43b4030b8306a10e940c976"
I20260619 16:46:16.618122 5418 raft_consensus.cc:3037] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.618206 5552 raft_consensus.cc:3037] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:16.619267 5418 raft_consensus.cc:2454] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645d980798884b11a7bd027e54d84f4e in term 1.
I20260619 16:46:16.619278 5552 raft_consensus.cc:2454] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645d980798884b11a7bd027e54d84f4e in term 1.
I20260619 16:46:16.619683 5212 leader_election.cc:302] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 645d980798884b11a7bd027e54d84f4e, c2606ae1e43b4030b8306a10e940c976; no voters:
I20260619 16:46:16.619892 5633 raft_consensus.cc:2781] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:16.620103 5633 raft_consensus.cc:686] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [term 1 LEADER]: Becoming Leader. State: Replica: 645d980798884b11a7bd027e54d84f4e, State: Running, Role: LEADER
I20260619 16:46:16.620380 5633 consensus_queue.cc:237] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } }
I20260619 16:46:16.621582 5668 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: TxnStatusTablet state changed. Reason: New leader 645d980798884b11a7bd027e54d84f4e. Latest consensus state: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.621699 5668 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:16.621964 5670 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:46:16.622133 5670 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:46:16.622888 5120 catalog_manager.cc:5495] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 645d980798884b11a7bd027e54d84f4e (127.3.233.129). New cstate: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:16.654846 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.650310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.3.233.129:38641)
0619 16:46:16.650445 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.650495 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.650535 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:46:16.650739 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.650741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.650796 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.650884 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:16.651203 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.651716 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.653121 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.653369 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.653537 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.653919 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:16.654385 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:16.654551 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.654644 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.654701 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:46:16.654899 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.650191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:38641 (local address 127.0.0.1:56416)
0619 16:46:16.650469 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.650485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.650608 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.650958 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.650962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.650970 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:16.651099 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.651106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.651832 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.651836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.652967 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.652980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.653572 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.653577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.653683 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.653749 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:16.654530 (+ 781us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:16.654548 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.654635 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.654740 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 16:46:16.658121 5418 raft_consensus.cc:1261] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Refusing update from remote peer 645d980798884b11a7bd027e54d84f4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:16.658433 5552 raft_consensus.cc:1261] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Refusing update from remote peer 645d980798884b11a7bd027e54d84f4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:16.658679 5668 consensus_queue.cc:1035] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:16.659042 5633 consensus_queue.cc:1035] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:16.660678 5634 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: TxnStatusTablet state changed. Reason: New leader 645d980798884b11a7bd027e54d84f4e. Latest consensus state: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.660808 5634 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:16.660826 5645 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: TxnStatusTablet state changed. Reason: New leader 645d980798884b11a7bd027e54d84f4e. Latest consensus state: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.660960 5645 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:16.663717 5668 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.663839 5668 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:16.663955 5668 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.664031 5668 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P 645d980798884b11a7bd027e54d84f4e: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:16.664273 5645 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.664402 5645 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P c2606ae1e43b4030b8306a10e940c976: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:16.664933 5640 mvcc.cc:205] Tried to move back new op lower bound from 7298611513985855488 to 7298611513839464448. Current Snapshot: MvccSnapshot[applied={T|T < 7298611513985855488 or (T in {7298611513985855488})}]
I20260619 16:46:16.666846 5634 tablet_replica.cc:439] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "645d980798884b11a7bd027e54d84f4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 } } peers { permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 } } peers { permanent_uuid: "c2606ae1e43b4030b8306a10e940c976" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44911 } } }
I20260619 16:46:16.666998 5634 tablet_replica.cc:442] T fe349d9cecf94f158c6ef84325e8fc1f P 6c66fc51eca543c5a71496269d53a8e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:16.681152 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.677481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:47301 (local address 127.3.233.129:38641)
0619 16:46:16.677622 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.677625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.677641 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:16.677794 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.677797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.677848 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.677934 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:16.678227 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.678755 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.680036 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.680277 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.680362 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.680705 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:16.680787 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:16.680859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.680981 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.681033 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:46:16.681186 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.677297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:38641 (local address 127.3.233.131:47301)
0619 16:46:16.677542 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.677566 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.677657 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.677963 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.677966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.677975 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:16.678089 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.678095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.678880 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.678884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.679875 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.679886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.680391 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.680394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.680481 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.680538 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:16.680881 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:16.680888 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.680954 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.681046 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 16:46:16.688633 5418 raft_consensus.cc:1261] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Refusing update from remote peer c2606ae1e43b4030b8306a10e940c976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:16.688664 5277 raft_consensus.cc:1261] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Refusing update from remote peer c2606ae1e43b4030b8306a10e940c976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:16.689090 5645 consensus_queue.cc:1035] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c66fc51eca543c5a71496269d53a8e9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:16.689284 5637 consensus_queue.cc:1035] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "645d980798884b11a7bd027e54d84f4e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:16.714839 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.710729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:51931 (local address 127.3.233.131:44911)
0619 16:46:16.710907 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.710912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.711074 (+ 162us) server_negotiation.cc:407] Connection header received
0619 16:46:16.711204 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.711206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.711245 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.711305 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:16.711597 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.712152 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.713670 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.714023 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.714126 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.714426 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:16.714498 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:16.714555 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.714679 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.714725 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:46:16.714946 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.710667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:44911 (local address 127.3.233.129:51931)
0619 16:46:16.711002 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:16.711018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:16.711125 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:16.711335 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:16.711338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:16.711345 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:16.711473 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.711479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.712272 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.712276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.713294 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:16.713305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.714137 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.714141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:16.714241 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.714298 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:16.714595 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:16.714601 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:46:16.714673 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:46:16.714757 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260619 16:46:16.736331 5121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:16.736517 5121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:16.738821 5121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95ff4d5b5ba64b8d838d0fe726d17057: Sending DeleteTablet for 3 replicas of tablet f36b9b6d4619437e8f815b444ab9141d
I20260619 16:46:16.739583 5258 tablet_service.cc:1514] Processing DeleteTablet for tablet f36b9b6d4619437e8f815b444ab9141d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56400
I20260619 16:46:16.739614 5398 tablet_service.cc:1514] Processing DeleteTablet for tablet f36b9b6d4619437e8f815b444ab9141d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:44256
I20260619 16:46:16.739799 5652 tablet_replica.cc:330] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: stopping tablet replica
I20260619 16:46:16.739809 5651 tablet_replica.cc:330] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: stopping tablet replica
I20260619 16:46:16.739938 5652 raft_consensus.cc:2229] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:16.739956 5651 raft_consensus.cc:2229] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:16.740099 5652 raft_consensus.cc:2258] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:16.740113 5651 raft_consensus.cc:2258] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:16.740590 5534 tablet_service.cc:1514] Processing DeleteTablet for tablet f36b9b6d4619437e8f815b444ab9141d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50352
I20260619 16:46:16.740779 5650 tablet_replica.cc:330] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: stopping tablet replica
I20260619 16:46:16.740882 5650 raft_consensus.cc:2229] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:16.740972 5651 ts_tablet_manager.cc:1905] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:16.740978 5652 ts_tablet_manager.cc:1905] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:16.741076 5650 raft_consensus.cc:2258] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:16.741783 5650 ts_tablet_manager.cc:1905] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:16.743906 5651 ts_tablet_manager.cc:1918] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:16.743975 5651 log.cc:1196] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/f36b9b6d4619437e8f815b444ab9141d
I20260619 16:46:16.743963 5652 ts_tablet_manager.cc:1918] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:16.744031 5652 log.cc:1196] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/f36b9b6d4619437e8f815b444ab9141d
I20260619 16:46:16.744222 5651 ts_tablet_manager.cc:1939] T f36b9b6d4619437e8f815b444ab9141d P 6c66fc51eca543c5a71496269d53a8e9: Deleting consensus metadata
I20260619 16:46:16.744297 5652 ts_tablet_manager.cc:1939] T f36b9b6d4619437e8f815b444ab9141d P 645d980798884b11a7bd027e54d84f4e: Deleting consensus metadata
I20260619 16:46:16.744611 5650 ts_tablet_manager.cc:1918] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:16.744678 5650 log.cc:1196] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/f36b9b6d4619437e8f815b444ab9141d
I20260619 16:46:16.744920 5106 catalog_manager.cc:4841] TS 6c66fc51eca543c5a71496269d53a8e9 (127.3.233.130:39655): tablet f36b9b6d4619437e8f815b444ab9141d (table test-table [id=64689f59a7f34a71893530426808cbea]) successfully deleted
I20260619 16:46:16.745110 5106 catalog_manager.cc:4841] TS 645d980798884b11a7bd027e54d84f4e (127.3.233.129:38641): tablet f36b9b6d4619437e8f815b444ab9141d (table test-table [id=64689f59a7f34a71893530426808cbea]) successfully deleted
I20260619 16:46:16.746219 5650 ts_tablet_manager.cc:1939] T f36b9b6d4619437e8f815b444ab9141d P c2606ae1e43b4030b8306a10e940c976: Deleting consensus metadata
I20260619 16:46:16.747175 5108 catalog_manager.cc:4841] TS c2606ae1e43b4030b8306a10e940c976 (127.3.233.131:44911): tablet f36b9b6d4619437e8f815b444ab9141d (table test-table [id=64689f59a7f34a71893530426808cbea]) successfully deleted
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:16.756865 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.746291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.3.233.129:38641)
0619 16:46:16.746424 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.746428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.746839 (+ 411us) server_negotiation.cc:407] Connection header received
0619 16:46:16.746990 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.746994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.747053 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.747137 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.747989 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.748543 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.749389 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.749614 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.749727 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.752917 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.752927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.752942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.752973 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.755003 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.755457 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.755466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.755469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.755583 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.755839 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.755844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.755847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.756186 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.756428 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.756529 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.756654 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 16:46:16.757382 5318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56418
I20260619 16:46:16.767740 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:16.759249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.3.233.190:33875)
0619 16:46:16.759376 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:16.759379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:16.759582 (+ 203us) server_negotiation.cc:407] Connection header received
0619 16:46:16.759664 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:16.759668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:16.759742 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:16.759822 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:16.760730 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.761258 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.762364 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:16.762592 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:16.762677 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:16.764355 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:16.764358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:16.764360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:16.764385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:16.766021 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.766501 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.766504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.766505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.766578 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:16.766848 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:16.766850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:16.766851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:16.767087 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:16.767200 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:16.767476 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:46:16.767575 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:46:16.768225 5121 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47794
I20260619 16:46:16.769629 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 5193
I20260619 16:46:16.777406 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 5333
I20260619 16:46:16.783569 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 5473
I20260619 16:46:16.790835 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 5089
2026-06-19T16:46:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2653 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5689](info): set up 2 sockets
Jun 19 16:46:16 dist-test-slave-cgwx krb5kdc[5689](info): commencing operation
krb5kdc: starting...
W20260619 16:46:17.842798 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:17 dist-test-slave-cgwx krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:17.970032 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34895
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:18.081007 5702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:18.081290 5702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:18.081331 5702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:18.085486 5702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:18.085556 5702 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:46:18.085573 5702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:18.085592 5702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:18.085608 5702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:18.085644 5702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:18.091054 5702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:18.092152 5702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:18.093192 5702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:18.101518 5708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.101614 5707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:18.101841 5702 server_base.cc:1034] running on GCE node
W20260619 16:46:18.101605 5710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:18.102488 5702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:18.103703 5702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:18.104876 5702 hybrid_clock.cc:648] HybridClock initialized: now 1781887578104854 us; error 36 us; skew 500 ppm
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:18.108052 5702 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:18.109028 5702 webserver.cc:460] Webserver started at http://127.3.233.190:43767/ using document root <none> and password file <none>
I20260619 16:46:18.109632 5702 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:18.109725 5702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:18.109969 5702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:18.112259 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "8ad6865ac6ed496c80e1d99a4daa5394"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "5d107d3946ee17c7744df3b05c5b81e8"
server_key_iv: "6074cdbc172d6cd46491111ae64225a8"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.112877 5702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "8ad6865ac6ed496c80e1d99a4daa5394"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "5d107d3946ee17c7744df3b05c5b81e8"
server_key_iv: "6074cdbc172d6cd46491111ae64225a8"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.117039 5702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:18.119758 5717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:18.121148 5702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:18.121310 5702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "8ad6865ac6ed496c80e1d99a4daa5394"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "5d107d3946ee17c7744df3b05c5b81e8"
server_key_iv: "6074cdbc172d6cd46491111ae64225a8"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.121438 5702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:18.135643 5702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:18.139380 5702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:18.139611 5702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:18.148169 5702 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34895
I20260619 16:46:18.148190 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34895 every 8 connection(s)
I20260619 16:46:18.149247 5702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:18.152227 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:18.156921 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 5702
I20260619 16:46:18.157022 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:18.157325 4006 external_mini_cluster.cc:1428] Setting key 773a57136cc43ded5e67d99a7671abc2
I20260619 16:46:18.158128 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Bootstrap starting.
I20260619 16:46:18.160838 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:18.161711 5780 log.cc:824] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:18.163892 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: No bootstrap required, opened a new log
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:18.168143 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34895 } }
I20260619 16:46:18.168442 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:18.168509 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad6865ac6ed496c80e1d99a4daa5394, State: Initialized, Role: FOLLOWER
I20260619 16:46:18.169064 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34895 } }
I20260619 16:46:18.169242 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:18.169332 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:18.169461 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:18.170710 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34895 } }
I20260619 16:46:18.171129 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ad6865ac6ed496c80e1d99a4daa5394; no voters:
I20260619 16:46:18.171511 5780 leader_election.cc:288] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:18.171579 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:18.171794 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad6865ac6ed496c80e1d99a4daa5394, State: Running, Role: LEADER
I20260619 16:46:18.172107 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34895 } }
I20260619 16:46:18.172852 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:18.173282 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ad6865ac6ed496c80e1d99a4daa5394. Latest consensus state: current_term: 1 leader_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34895 } } }
I20260619 16:46:18.173384 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:18.174489 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.158817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.3.233.190:34895)
0619 16:46:18.159376 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.159397 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.159446 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:46:18.160285 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.160309 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.160730 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.161140 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.161906 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.163275 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.164353 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.165052 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.165209 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.168222 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.168228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.168246 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.168280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.170522 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.171046 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.171053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.171059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.171207 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.171679 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.171684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.171685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.172270 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.172429 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.172966 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.173330 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":196,"threads_started":1}
I20260619 16:46:18.175526 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad6865ac6ed496c80e1d99a4daa5394" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34895 } } }
I20260619 16:46:18.176342 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394 [sys.catalog]: This master's current role is: LEADER
W20260619 16:46:18.177989 5800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:18.178066 5800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:18.178184 5789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:18.179116 5789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:18.185398 5789 catalog_manager.cc:1347] Generated new cluster ID: a271334a13fb4d4fab896efa2bbcbba8
I20260619 16:46:18.185477 5789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:18.203462 5789 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:18.204777 5789 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:18.212857 5789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Generated new TSK 0
I20260619 16:46:18.213688 5789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:18.249285 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34895
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:18.362198 5806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:18.362488 5806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:18.362538 5806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:18.366694 5806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:18.366758 5806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:18.366801 5806 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:46:18.366849 5806 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:18.366938 5806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:18.372599 5806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:18.373783 5806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:18.374994 5806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:18.384654 5811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.384702 5814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.385026 5812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:18.385072 5806 server_base.cc:1034] running on GCE node
I20260619 16:46:18.385460 5806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:18.386153 5806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:18.387336 5806 hybrid_clock.cc:648] HybridClock initialized: now 1781887578387314 us; error 33 us; skew 500 ppm
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:18.390120 5806 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:18.391065 5806 webserver.cc:460] Webserver started at http://127.3.233.129:45589/ using document root <none> and password file <none>
I20260619 16:46:18.391664 5806 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:18.391747 5806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:18.391965 5806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:18.393963 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "2dc59c5e80204b9bad858d983ecfa36f"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "c66455b5023faf6041c507159950ea71"
server_key_iv: "cef4e8936cf26fa04df6eec420f70937"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.394519 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "2dc59c5e80204b9bad858d983ecfa36f"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "c66455b5023faf6041c507159950ea71"
server_key_iv: "cef4e8936cf26fa04df6eec420f70937"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.398244 5806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 16:46:18.400614 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:18.401850 5806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:18.401984 5806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "2dc59c5e80204b9bad858d983ecfa36f"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "c66455b5023faf6041c507159950ea71"
server_key_iv: "cef4e8936cf26fa04df6eec420f70937"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.402117 5806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:18.418553 5806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:18.421725 5806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:18.421939 5806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:18.423542 5806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:18.423617 5806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:18.423705 5806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:18.423743 5806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:18.434737 5806 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:42457
I20260619 16:46:18.434852 5936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:42457 every 8 connection(s)
I20260619 16:46:18.435905 5806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:18.437381 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.424306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:58927 (local address 127.3.233.190:34895)
0619 16:46:18.424452 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.424457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.425934 (+ 1477us) server_negotiation.cc:407] Connection header received
0619 16:46:18.427439 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.427442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.427595 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.427710 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.428946 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.429476 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.430614 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.430847 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.430977 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.433495 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.433499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.433502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.433531 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.435256 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.435835 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.435838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.435839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.435920 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.436323 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.436327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.436329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.436562 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.436672 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.437152 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.437261 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:18.438261 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.425235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34895 (local address 127.3.233.129:58927)
0619 16:46:18.425767 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.425825 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.427180 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.427897 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.427910 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.428320 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:18.428808 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.428821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.429596 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.429599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.430492 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.430501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.431032 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.431036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.431198 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.431823 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:18.431839 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:18.433302 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:18.435362 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.435367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.435382 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.435719 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.436039 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.436044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.436046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.436214 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.436682 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:18.436688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:18.436939 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.437097 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.437379 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260619 16:46:18.439759 5937 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34895
I20260619 16:46:18.439934 5937 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:18.440536 5937 heartbeater.cc:511] Master 127.3.233.190:34895 requested a full tablet report, sending...
I20260619 16:46:18.442235 5733 ts_manager.cc:194] Registered new tserver with Master: 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457)
I20260619 16:46:18.443639 5733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:58927
I20260619 16:46:18.445760 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 5806
I20260619 16:46:18.445909 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:18.446182 4006 external_mini_cluster.cc:1428] Setting key ec4e7f9f2815854a6bef2d3fb37ac05b
I20260619 16:46:18.450029 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.442114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:41759 (local address 127.3.233.190:34895)
0619 16:46:18.442279 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.442282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.442457 (+ 175us) server_negotiation.cc:407] Connection header received
0619 16:46:18.442533 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.442535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.442575 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.442646 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.443231 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.443756 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.444741 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.444962 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.445065 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.446313 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.446317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.446320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.446347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.448051 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.448584 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.448591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.448596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.448691 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.449051 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.449058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.449062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.449318 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.449424 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.449762 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.449867 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 16:46:18.450040 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.441992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34895 (local address 127.3.233.129:41759)
0619 16:46:18.442307 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.442323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.442440 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.442679 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.442681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.442952 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:18.443117 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.443125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.443894 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.443897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.444591 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.444600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.445073 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.445077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.445168 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.445719 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:18.445746 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:18.446180 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:18.448165 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.448168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.448170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.448413 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.448827 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.448830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.448832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.448950 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.449479 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:18.449485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:18.449599 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.449709 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.449862 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:18.474876 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34895
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:18.595541 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:18.595808 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:18.595883 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:18.599948 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:18.600014 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:18.600086 5946 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:46:18.600117 5946 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:18.600201 5946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:18.605809 5946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:18.606905 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:18.608095 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:18.617939 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.617960 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.617964 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:18.618312 5946 server_base.cc:1034] running on GCE node
I20260619 16:46:18.618738 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:18.619501 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:18.620689 5946 hybrid_clock.cc:648] HybridClock initialized: now 1781887578620667 us; error 42 us; skew 500 ppm
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:18.623958 5946 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:18.624925 5946 webserver.cc:460] Webserver started at http://127.3.233.130:40325/ using document root <none> and password file <none>
I20260619 16:46:18.625680 5946 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:18.625782 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:18.626024 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:18.628369 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "9931f0d5e4113c837d31ead47b5ff81f"
server_key_iv: "46fd09be87b7367e854be182b3ec348e"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.629012 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "9931f0d5e4113c837d31ead47b5ff81f"
server_key_iv: "46fd09be87b7367e854be182b3ec348e"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.633234 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:18.635870 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:18.637310 5946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:18.637476 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "9931f0d5e4113c837d31ead47b5ff81f"
server_key_iv: "46fd09be87b7367e854be182b3ec348e"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.637605 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:18.659864 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:18.662884 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:18.663120 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:18.664645 5946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:18.664714 5946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:18.664785 5946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:18.664827 5946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:18.676568 5946 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:34713
I20260619 16:46:18.676822 6076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:34713 every 8 connection(s)
I20260619 16:46:18.677627 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:18.679692 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.665586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:52053 (local address 127.3.233.190:34895)
0619 16:46:18.665734 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.665738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.666872 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 16:46:18.668167 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.668170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.668234 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.668345 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.669777 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.670358 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.671813 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.672202 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.672330 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.675562 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.675567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.675570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.675605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.677325 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.677958 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.677962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.677963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.678092 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.678559 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.678562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.678564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.678776 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.678858 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.679443 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.679565 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:46:18.680420 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.666130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34895 (local address 127.3.233.130:52053)
0619 16:46:18.666699 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.666746 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.667925 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.668532 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.668553 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.668994 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:18.669624 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.669639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.670491 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.670494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.671685 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.671695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.672381 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.672386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.672568 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.673186 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:18.673202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:18.675367 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:18.677449 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.677455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.677468 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.677812 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.678240 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.678243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.678245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.678437 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.678870 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:18.678875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:18.679168 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.679321 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.679567 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":140,"threads_started":1}
I20260619 16:46:18.681689 6077 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34895
I20260619 16:46:18.681838 6077 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:18.682246 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 5946
I20260619 16:46:18.682302 6077 heartbeater.cc:511] Master 127.3.233.190:34895 requested a full tablet report, sending...
I20260619 16:46:18.682363 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:18.682718 4006 external_mini_cluster.cc:1428] Setting key b31bdaffce3b16a9571bc0fe5175d235
I20260619 16:46:18.683370 5734 ts_manager.cc:194] Registered new tserver with Master: 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713)
I20260619 16:46:18.684036 5734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:52053
I20260619 16:46:18.691995 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.684649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:50137 (local address 127.3.233.190:34895)
0619 16:46:18.684778 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.684785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.684879 (+ 94us) server_negotiation.cc:407] Connection header received
0619 16:46:18.685021 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.685024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.685067 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.685156 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.685761 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.686362 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.687322 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.687532 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.687633 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.688879 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.688883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.688886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.688911 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.690338 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.690740 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.690745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.690747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.690821 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.691115 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.691119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.691121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.691366 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.691474 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.691733 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.691838 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:46:18.692008 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.684355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34895 (local address 127.3.233.130:50137)
0619 16:46:18.684651 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.684668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.684927 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.685188 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.685195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.685454 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:18.685631 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.685642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.686486 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.686489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.687214 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.687223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.687635 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.687638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.687736 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.688388 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:18.688399 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:18.688754 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:18.690444 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.690449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.690451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.690647 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.690915 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.690939 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.690942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.691026 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.691462 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:18.691465 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:18.691561 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.691678 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.691827 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:18.711159 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34895
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:18.822970 6086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:18.823246 6086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:18.823318 6086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:18.827402 6086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:18.827461 6086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:18.827504 6086 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:46:18.827553 6086 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:18.827615 6086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:18.832863 6086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:18.834052 6086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:18.835109 6086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:18.845638 6091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.845654 6092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:18.845662 6094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:18.846026 6086 server_base.cc:1034] running on GCE node
I20260619 16:46:18.846448 6086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:18.847251 6086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:18.848515 6086 hybrid_clock.cc:648] HybridClock initialized: now 1781887578848495 us; error 31 us; skew 500 ppm
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:18.851549 6086 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:18.852547 6086 webserver.cc:460] Webserver started at http://127.3.233.131:45409/ using document root <none> and password file <none>
I20260619 16:46:18.853137 6086 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:18.853225 6086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:18.853447 6086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:18.855613 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "f006ce00276543818e4bcc9f2aebada5"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "9690965e28cf9f81562c24d9d1f4409f"
server_key_iv: "e0820ad332ddb7d3344e51f44f56003a"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.856186 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "f006ce00276543818e4bcc9f2aebada5"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "9690965e28cf9f81562c24d9d1f4409f"
server_key_iv: "e0820ad332ddb7d3344e51f44f56003a"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.860136 6086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 16:46:18.862708 6101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:18.863993 6086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:18.864131 6086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "f006ce00276543818e4bcc9f2aebada5"
format_stamp: "Formatted at 2026-06-19 16:46:18 on dist-test-slave-cgwx"
server_key: "9690965e28cf9f81562c24d9d1f4409f"
server_key_iv: "e0820ad332ddb7d3344e51f44f56003a"
server_key_version: "encryptionkey@0"
I20260619 16:46:18.864246 6086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:18.880491 6086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:18.883842 6086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:18.884047 6086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:18.885545 6086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:18.885622 6086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:18.885687 6086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:18.885740 6086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:18.897217 6086 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:38911
I20260619 16:46:18.898151 6086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:18.898111 6216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:38911 every 8 connection(s)
I20260619 16:46:18.900056 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.886260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:59495 (local address 127.3.233.190:34895)
0619 16:46:18.886382 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.886386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.888071 (+ 1685us) server_negotiation.cc:407] Connection header received
0619 16:46:18.889233 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.889237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.889294 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.889395 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.890969 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.891625 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.893017 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.893285 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.893398 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.896127 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.896131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.896134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.896163 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.897863 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.898472 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.898477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.898479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.898584 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.899025 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.899028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.899029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.899267 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.899348 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.899828 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.899935 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:46:18.900857 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.887403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34895 (local address 127.3.233.131:59495)
0619 16:46:18.887918 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.887961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.889013 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.889564 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.889571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.890047 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:18.890793 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.890816 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.891784 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.891787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.892895 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.892905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.893484 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.893491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.893664 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.894291 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:18.894307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:18.895930 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:18.897974 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.897981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.897998 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.898372 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.898696 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.898699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.898702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.898885 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.899367 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:18.899373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:18.899605 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.899764 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.900021 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260619 16:46:18.902040 6217 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34895
I20260619 16:46:18.902179 6217 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:18.902732 6217 heartbeater.cc:511] Master 127.3.233.190:34895 requested a full tablet report, sending...
I20260619 16:46:18.903832 5733 ts_manager.cc:194] Registered new tserver with Master: f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911)
I20260619 16:46:18.904444 5733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:59495
I20260619 16:46:18.907780 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 6086
I20260619 16:46:18.907933 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:18.908244 4006 external_mini_cluster.cc:1428] Setting key bcbabc7402e5b5ab7c060ef3fbde6ab5
I20260619 16:46:18.910310 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:18.912176 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.904354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:59061 (local address 127.3.233.190:34895)
0619 16:46:18.904465 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.904469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.904809 (+ 340us) server_negotiation.cc:407] Connection header received
0619 16:46:18.904942 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.904945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.904986 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.905068 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.905499 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.906129 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.906961 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.907182 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.907287 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.908557 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.908560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.908562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.908582 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.910386 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.910870 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.910873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.910875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.911011 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.911337 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.911339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.911340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.911513 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.911592 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.911962 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.912065 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:46:18.912297 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.904457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34895 (local address 127.3.233.131:59061)
0619 16:46:18.904721 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.904737 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.904849 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.905086 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.905089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.905298 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:18.905395 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.905401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.906246 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.906250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.906825 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.906832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.907287 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.907291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.907386 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.907943 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:18.907973 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:18.908436 (+ 463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:18.910502 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.910506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.910508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.910744 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.911137 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:18.911142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:18.911143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:18.911244 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:18.911606 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:18.911612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:18.911730 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.911842 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.912019 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:46:18 dist-test-slave-cgwx krb5kdc[5689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:18.938329 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.929380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.3.233.190:34895)
0619 16:46:18.929505 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.929509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.929520 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:46:18.929558 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.929560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.929596 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.929667 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:18.930485 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.931178 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.932119 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.932340 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.932425 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.935075 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:18.935078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:18.935080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:18.935106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:18.936577 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.937110 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.937113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.937114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.937190 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:18.937477 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:18.937480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:18.937482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:18.937714 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:18.937819 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.938069 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.938165 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:46:18.941506 5733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:18.944758 5733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:18.961488 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.954152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38911 (local address 127.0.0.1:34946)
0619 16:46:18.955010 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.955025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.955113 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.956045 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.956047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.956065 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:18.956164 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.956170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.957614 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.957617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.959486 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.959496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.960744 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.960746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.961115 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.961142 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.961218 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.961325 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":207,"thread_start_us":104,"threads_started":1}
I20260619 16:46:18.961594 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.954246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.3.233.129:42457)
0619 16:46:18.954582 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.954589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.955233 (+ 644us) server_negotiation.cc:407] Connection header received
0619 16:46:18.955326 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.955332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.955561 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.955721 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:18.956050 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.957276 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.959512 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.960422 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.960534 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.960689 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.961309 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.961428 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
I20260619 16:46:18.962101 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.954582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.3.233.131:38911)
0619 16:46:18.954946 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.954960 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.955120 (+ 160us) server_negotiation.cc:407] Connection header received
0619 16:46:18.955259 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.955293 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.955600 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.955876 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:18.956317 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.957487 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.959634 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.960427 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.960525 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.960683 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.961811 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.961911 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
I20260619 16:46:18.961971 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.953813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42457 (local address 127.0.0.1:33764)
0619 16:46:18.954670 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.954686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.954789 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.955725 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.955731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.955757 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:18.955905 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.955912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.957404 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.957407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.959378 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.959387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.960755 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.960759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.960998 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.961061 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.961178 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.961821 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":567,"thread_start_us":91,"threads_started":1}
I20260619 16:46:18.963609 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.953419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34713 (local address 127.0.0.1:50644)
0619 16:46:18.953873 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:18.953944 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:18.954140 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:18.957706 (+ 3566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:18.957709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:18.958306 (+ 597us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:18.958418 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.958425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.960739 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.960742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.961866 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:18.961875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.963267 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.963271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:18.963355 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.963380 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:46:18.963428 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:46:18.963490 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":30,"thread_start_us":94,"threads_started":1}
I20260619 16:46:18.964444 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:18.956069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.3.233.130:34713)
0619 16:46:18.957227 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:18.957239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:18.957261 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:46:18.957338 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:18.957343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:18.957538 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:18.957948 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:18.959534 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.960620 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.962232 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:18.963132 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:18.964042 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:18.964182 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:18.964230 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 16:46:18.964309 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":112,"threads_started":1}
I20260619 16:46:18.964720 5871 tablet_service.cc:1467] Processing CreateTablet for tablet e8651dbf9f4d42a687910e854a685b9d (DEFAULT_TABLE table=test-table [id=fe68da6ef6c345a9bf743c4a80683f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:18.964918 6151 tablet_service.cc:1467] Processing CreateTablet for tablet e8651dbf9f4d42a687910e854a685b9d (DEFAULT_TABLE table=test-table [id=fe68da6ef6c345a9bf743c4a80683f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:18.965582 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8651dbf9f4d42a687910e854a685b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:18.965865 6151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8651dbf9f4d42a687910e854a685b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:18.966980 6011 tablet_service.cc:1467] Processing CreateTablet for tablet e8651dbf9f4d42a687910e854a685b9d (DEFAULT_TABLE table=test-table [id=fe68da6ef6c345a9bf743c4a80683f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:18.967923 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8651dbf9f4d42a687910e854a685b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:18.972194 6240 tablet_bootstrap.cc:492] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Bootstrap starting.
I20260619 16:46:18.973795 6241 tablet_bootstrap.cc:492] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Bootstrap starting.
I20260619 16:46:18.974795 6240 tablet_bootstrap.cc:654] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:18.975246 6242 tablet_bootstrap.cc:492] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Bootstrap starting.
I20260619 16:46:18.975762 6240 log.cc:824] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:18.976018 6241 tablet_bootstrap.cc:654] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:18.976912 6241 log.cc:824] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:18.977380 6242 tablet_bootstrap.cc:654] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:18.977845 6240 tablet_bootstrap.cc:492] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: No bootstrap required, opened a new log
I20260619 16:46:18.978048 6240 ts_tablet_manager.cc:1397] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:46:18.978070 6242 log.cc:824] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:18.980118 6241 tablet_bootstrap.cc:492] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: No bootstrap required, opened a new log
I20260619 16:46:18.980320 6241 ts_tablet_manager.cc:1397] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:46:18.982784 6240 raft_consensus.cc:348] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:18.983126 6240 raft_consensus.cc:374] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:18.983193 6240 raft_consensus.cc:729] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc59c5e80204b9bad858d983ecfa36f, State: Initialized, Role: FOLLOWER
I20260619 16:46:18.983673 6240 consensus_queue.cc:260] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:18.984288 6241 raft_consensus.cc:348] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:18.984534 6241 raft_consensus.cc:374] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:18.984586 6241 raft_consensus.cc:729] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Initialized, Role: FOLLOWER
I20260619 16:46:18.984722 6240 ts_tablet_manager.cc:1428] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:46:18.984771 6242 tablet_bootstrap.cc:492] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: No bootstrap required, opened a new log
I20260619 16:46:18.985005 6242 ts_tablet_manager.cc:1397] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260619 16:46:18.984993 6241 consensus_queue.cc:260] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:18.985105 5937 heartbeater.cc:503] Master 127.3.233.190:34895 was elected leader, sending a full tablet report...
I20260619 16:46:18.985622 6217 heartbeater.cc:503] Master 127.3.233.190:34895 was elected leader, sending a full tablet report...
I20260619 16:46:18.985975 6241 ts_tablet_manager.cc:1428] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:46:18.989339 6242 raft_consensus.cc:348] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:18.989662 6242 raft_consensus.cc:374] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:18.989754 6242 raft_consensus.cc:729] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6591d76ee4490aafb8eefb7ab198e1, State: Initialized, Role: FOLLOWER
I20260619 16:46:18.990231 6242 consensus_queue.cc:260] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:18.990893 6077 heartbeater.cc:503] Master 127.3.233.190:34895 was elected leader, sending a full tablet report...
I20260619 16:46:18.991333 6242 ts_tablet_manager.cc:1428] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260619 16:46:19.015403 6247 raft_consensus.cc:482] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:19.015713 6247 raft_consensus.cc:504] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:19.017261 6247 leader_election.cc:288] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713), 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457)
I20260619 16:46:19.021446 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.017801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:47069 (local address 127.3.233.130:34713)
0619 16:46:19.017940 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.017945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.018222 (+ 277us) server_negotiation.cc:407] Connection header received
0619 16:46:19.018316 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.018318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.018366 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.018458 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:19.018712 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.019273 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.020333 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.020899 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.020988 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.021030 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.021260 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.021313 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:19.021493 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.017438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34713 (local address 127.3.233.131:47069)
0619 16:46:19.018128 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.018143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.018237 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.018480 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.018483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.018498 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:19.018612 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.018618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.019378 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.019380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.020212 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.020221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.021013 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.021016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.021142 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.021168 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.021247 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.021314 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":273,"thread_start_us":136,"threads_started":1}
I20260619 16:46:19.021886 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.017438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42457 (local address 127.3.233.131:44751)
0619 16:46:19.017593 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.017612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.017717 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.018249 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.018253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.018277 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:19.018402 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.018412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.019277 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.019280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.020631 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.020640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.021432 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.021436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.021551 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.021580 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.021685 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.021773 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260619 16:46:19.021904 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.017887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:44751 (local address 127.3.233.129:42457)
0619 16:46:19.017995 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.017999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.018014 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:19.018067 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.018070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.018118 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.018209 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:19.018514 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.019181 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.020765 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.021327 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.021434 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.021501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.021710 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.021783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:46:19.022122 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8651dbf9f4d42a687910e854a685b9d" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" is_pre_election: true
I20260619 16:46:19.022445 6031 raft_consensus.cc:2454] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 0.
I20260619 16:46:19.022519 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8651dbf9f4d42a687910e854a685b9d" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc59c5e80204b9bad858d983ecfa36f" is_pre_election: true
I20260619 16:46:19.022804 5891 raft_consensus.cc:2454] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 0.
I20260619 16:46:19.022900 6105 leader_election.cc:302] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6591d76ee4490aafb8eefb7ab198e1, f006ce00276543818e4bcc9f2aebada5; no voters:
I20260619 16:46:19.023255 6247 raft_consensus.cc:2781] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:19.023332 6247 raft_consensus.cc:482] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:19.023362 6247 raft_consensus.cc:3037] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.024315 6247 raft_consensus.cc:504] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:19.024693 6247 leader_election.cc:288] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713), 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457)
I20260619 16:46:19.025050 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8651dbf9f4d42a687910e854a685b9d" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
I20260619 16:46:19.025065 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8651dbf9f4d42a687910e854a685b9d" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc59c5e80204b9bad858d983ecfa36f"
I20260619 16:46:19.025153 6031 raft_consensus.cc:3037] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.025199 5891 raft_consensus.cc:3037] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.026204 6031 raft_consensus.cc:2454] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 1.
I20260619 16:46:19.026360 5891 raft_consensus.cc:2454] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 1.
I20260619 16:46:19.026547 6105 leader_election.cc:302] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6591d76ee4490aafb8eefb7ab198e1, f006ce00276543818e4bcc9f2aebada5; no voters:
I20260619 16:46:19.026750 6247 raft_consensus.cc:2781] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:19.026981 6247 raft_consensus.cc:686] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 1 LEADER]: Becoming Leader. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Running, Role: LEADER
I20260619 16:46:19.027369 6247 consensus_queue.cc:237] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } }
I20260619 16:46:19.030503 5734 catalog_manager.cc:5495] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f006ce00276543818e4bcc9f2aebada5 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:46:19 dist-test-slave-cgwx krb5kdc[5689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887578, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:19.097761 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.088753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34895 (local address 127.0.0.1:41234)
0619 16:46:19.089091 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.089110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.089244 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.089563 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.089565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.090006 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:19.090104 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.090110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.091057 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.091060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.091773 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.091783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.092235 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.092238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.092309 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.092875 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:19.092887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:19.094475 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:19.096226 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:19.096234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:19.096239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:19.096489 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:19.096753 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:19.096755 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:19.096756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:19.096845 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:19.097210 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:19.097216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:19.097329 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.097432 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.097561 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":145,"threads_started":1}
I20260619 16:46:19.097767 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.088873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.3.233.190:34895)
0619 16:46:19.089017 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.089021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.089213 (+ 192us) server_negotiation.cc:407] Connection header received
0619 16:46:19.089320 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.089322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.089365 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.089533 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:19.090230 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.090848 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.091906 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.092112 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.092206 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.094677 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:19.094680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:19.094682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:19.094702 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:19.096124 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:19.096578 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:19.096581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:19.096582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:19.096680 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:19.096918 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:19.096920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:19.096921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:19.097086 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:19.097159 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.097492 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.097617 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":53}
I20260619 16:46:19.101699 5734 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:41234:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:46:19.102497 5734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:19.108057 5871 tablet_service.cc:1467] Processing CreateTablet for tablet 93d19912ce11438abcc94ce091ec1e47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68c94c209ff9429b981291fad15e2420]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:19.108946 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d19912ce11438abcc94ce091ec1e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:19.111627 6151 tablet_service.cc:1467] Processing CreateTablet for tablet 93d19912ce11438abcc94ce091ec1e47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68c94c209ff9429b981291fad15e2420]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:19.112296 6151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d19912ce11438abcc94ce091ec1e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:19.112742 6240 tablet_bootstrap.cc:492] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: Bootstrap starting.
I20260619 16:46:19.113824 6240 tablet_bootstrap.cc:654] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:19.115273 6240 tablet_bootstrap.cc:492] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: No bootstrap required, opened a new log
I20260619 16:46:19.115375 6240 ts_tablet_manager.cc:1397] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:19.115538 6011 tablet_service.cc:1467] Processing CreateTablet for tablet 93d19912ce11438abcc94ce091ec1e47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68c94c209ff9429b981291fad15e2420]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:19.115885 6240 raft_consensus.cc:348] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.116014 6240 raft_consensus.cc:374] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:19.116071 6240 raft_consensus.cc:729] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc59c5e80204b9bad858d983ecfa36f, State: Initialized, Role: FOLLOWER
I20260619 16:46:19.116216 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d19912ce11438abcc94ce091ec1e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:19.116211 6240 consensus_queue.cc:260] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.116312 6241 tablet_bootstrap.cc:492] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: Bootstrap starting.
I20260619 16:46:19.116487 6240 ts_tablet_manager.cc:1428] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:19.116485 6246 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.116670 6246 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:19.117221 6241 tablet_bootstrap.cc:654] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:19.118577 6241 tablet_bootstrap.cc:492] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: No bootstrap required, opened a new log
I20260619 16:46:19.118644 6241 ts_tablet_manager.cc:1397] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:19.119210 6242 tablet_bootstrap.cc:492] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: Bootstrap starting.
I20260619 16:46:19.119194 6241 raft_consensus.cc:348] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.119324 6241 raft_consensus.cc:374] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:19.119374 6241 raft_consensus.cc:729] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Initialized, Role: FOLLOWER
I20260619 16:46:19.119536 6241 consensus_queue.cc:260] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.119890 6241 ts_tablet_manager.cc:1428] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:19.120205 6242 tablet_bootstrap.cc:654] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:19.120870 6247 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.121038 6247 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:19.121505 6242 tablet_bootstrap.cc:492] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: No bootstrap required, opened a new log
I20260619 16:46:19.121596 6242 ts_tablet_manager.cc:1397] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:19.122040 6242 raft_consensus.cc:348] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.122188 6242 raft_consensus.cc:374] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:19.122236 6242 raft_consensus.cc:729] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6591d76ee4490aafb8eefb7ab198e1, State: Initialized, Role: FOLLOWER
I20260619 16:46:19.122370 6242 consensus_queue.cc:260] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.122637 6242 ts_tablet_manager.cc:1428] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:19.122640 6248 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.122803 6248 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 16:46:19.149390 6218 tablet.cc:2369] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:19.178853 6078 tablet.cc:2369] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:19.187330 5938 tablet.cc:2369] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:19.328856 6247 raft_consensus.cc:482] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:19.329006 6247 raft_consensus.cc:504] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.329507 6247 leader_election.cc:288] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457), 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713)
I20260619 16:46:19.330118 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d19912ce11438abcc94ce091ec1e47" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" is_pre_election: true
I20260619 16:46:19.330118 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d19912ce11438abcc94ce091ec1e47" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc59c5e80204b9bad858d983ecfa36f" is_pre_election: true
I20260619 16:46:19.330302 6031 raft_consensus.cc:2454] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 0.
I20260619 16:46:19.330302 5891 raft_consensus.cc:2454] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 0.
I20260619 16:46:19.330673 6105 leader_election.cc:302] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dc59c5e80204b9bad858d983ecfa36f, 9f6591d76ee4490aafb8eefb7ab198e1, f006ce00276543818e4bcc9f2aebada5; no voters:
I20260619 16:46:19.330900 6247 raft_consensus.cc:2781] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:19.331018 6247 raft_consensus.cc:482] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:19.331082 6247 raft_consensus.cc:3037] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.332156 6247 raft_consensus.cc:504] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.332511 6247 leader_election.cc:288] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 election: Requested vote from peers 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457), 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713)
I20260619 16:46:19.332933 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d19912ce11438abcc94ce091ec1e47" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
I20260619 16:46:19.332933 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d19912ce11438abcc94ce091ec1e47" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc59c5e80204b9bad858d983ecfa36f"
I20260619 16:46:19.333079 6031 raft_consensus.cc:3037] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.333079 5891 raft_consensus.cc:3037] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.334342 6031 raft_consensus.cc:2454] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 1.
I20260619 16:46:19.334342 5891 raft_consensus.cc:2454] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 1.
I20260619 16:46:19.334698 6102 leader_election.cc:302] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dc59c5e80204b9bad858d983ecfa36f, 9f6591d76ee4490aafb8eefb7ab198e1, f006ce00276543818e4bcc9f2aebada5; no voters:
I20260619 16:46:19.334908 6247 raft_consensus.cc:2781] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:19.335073 6247 raft_consensus.cc:686] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [term 1 LEADER]: Becoming Leader. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Running, Role: LEADER
I20260619 16:46:19.335227 6247 consensus_queue.cc:237] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.336015 6250 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: TxnStatusTablet state changed. Reason: New leader f006ce00276543818e4bcc9f2aebada5. Latest consensus state: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.336141 6250 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:19.336475 6262 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:46:19.336766 6262 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:46:19.337286 5733 catalog_manager.cc:5495] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f006ce00276543818e4bcc9f2aebada5 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } health_report { overall_health: HEALTHY } } }
I20260619 16:46:19.357101 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.353220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38911 (local address 127.0.0.1:34954)
0619 16:46:19.353381 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.353399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.353512 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.353841 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.353844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.353851 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:19.353959 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.353965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.354759 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.354764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.355480 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.355490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.355957 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.355961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.356058 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.356108 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.356792 (+ 684us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.356810 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.356877 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.356969 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:46:19.357102 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.353343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.3.233.131:38911)
0619 16:46:19.353472 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.353477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.353491 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:19.353677 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.353679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.353732 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.353829 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:19.354068 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.354654 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.355610 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.355834 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.355954 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.356263 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.356697 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.356818 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.356896 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.356955 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:46:19.362807 5891 raft_consensus.cc:1261] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Refusing update from remote peer f006ce00276543818e4bcc9f2aebada5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:19.362813 6031 raft_consensus.cc:1261] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Refusing update from remote peer f006ce00276543818e4bcc9f2aebada5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:19.363607 6250 consensus_queue.cc:1035] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:19.363835 6247 consensus_queue.cc:1035] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:19.368072 6248 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: TxnStatusTablet state changed. Reason: New leader f006ce00276543818e4bcc9f2aebada5. Latest consensus state: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.368191 6248 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:19.368286 6246 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: TxnStatusTablet state changed. Reason: New leader f006ce00276543818e4bcc9f2aebada5. Latest consensus state: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.368436 6246 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:19.371340 6247 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.371464 6247 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:19.371773 6248 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.371872 6248 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P 9f6591d76ee4490aafb8eefb7ab198e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:19.372295 6247 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.372395 6247 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P f006ce00276543818e4bcc9f2aebada5: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:19.373222 6267 mvcc.cc:205] Tried to move back new op lower bound from 7298611525062774784 to 7298611524958568448. Current Snapshot: MvccSnapshot[applied={T|T < 7298611525062774784}]
I20260619 16:46:19.375699 6246 tablet_replica.cc:439] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } } }
I20260619 16:46:19.375818 6246 tablet_replica.cc:442] T 93d19912ce11438abcc94ce091ec1e47 P 2dc59c5e80204b9bad858d983ecfa36f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:19.385540 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.382105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.3.233.131:38911)
0619 16:46:19.382246 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.382252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.382266 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:19.382412 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.382415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.382475 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.382548 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:19.382799 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.383620 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.384485 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.384820 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.384919 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.385117 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.385201 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.385270 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.385342 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.385396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:46:19.397251 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.393705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:53995 (local address 127.3.233.131:38911)
0619 16:46:19.393886 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.393890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.393902 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:46:19.394018 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.394020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.394061 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.394126 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:19.394374 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.394892 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.396116 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.396348 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.396437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.396757 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.396823 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.396882 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.397045 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.397092 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 16:46:19.397254 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.393660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38911 (local address 127.3.233.131:53995)
0619 16:46:19.393823 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.393838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.393924 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.394146 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.394149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.394156 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:19.394278 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.394283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.395067 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.395071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.395972 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.395982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.396452 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.396456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.396549 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.396599 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.396902 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.396909 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.396978 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.397066 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":5}
I20260619 16:46:19.404845 6031 raft_consensus.cc:1261] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Refusing update from remote peer f006ce00276543818e4bcc9f2aebada5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:19.404876 5891 raft_consensus.cc:1261] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Refusing update from remote peer f006ce00276543818e4bcc9f2aebada5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:19.405256 6250 consensus_queue.cc:1035] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:19.405553 6250 consensus_queue.cc:1035] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:19.458248 5734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:19.458451 5734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:19.461155 5734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Sending DeleteTablet for 3 replicas of tablet e8651dbf9f4d42a687910e854a685b9d
I20260619 16:46:19.462007 6011 tablet_service.cc:1514] Processing DeleteTablet for tablet e8651dbf9f4d42a687910e854a685b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:19 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50644
I20260619 16:46:19.462006 5871 tablet_service.cc:1514] Processing DeleteTablet for tablet e8651dbf9f4d42a687910e854a685b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:19 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:33764
I20260619 16:46:19.462558 5734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:46:19.462987 6281 tablet_replica.cc:330] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: stopping tablet replica
W20260619 16:46:19.462993 5734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:19.463301 6280 tablet_replica.cc:330] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: stopping tablet replica
I20260619 16:46:19.463373 6281 raft_consensus.cc:2229] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:19.463613 6281 raft_consensus.cc:2258] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:19.463793 6280 raft_consensus.cc:2229] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:19.463773 6151 tablet_service.cc:1514] Processing DeleteTablet for tablet e8651dbf9f4d42a687910e854a685b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:19 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:34946
I20260619 16:46:19.464102 6280 raft_consensus.cc:2258] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:19.465323 6281 ts_tablet_manager.cc:1905] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:19.465346 6280 ts_tablet_manager.cc:1905] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:19.468711 6280 ts_tablet_manager.cc:1918] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:19.468780 6281 ts_tablet_manager.cc:1918] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:19.468858 5871 tablet_service.cc:1467] Processing CreateTablet for tablet 12985949eebe494a97d3e6c8425a5414 (DEFAULT_TABLE table=test-table [id=a85b0b7ee4394c1baddb9e16592ab38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:19.468935 6280 log.cc:1196] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/e8651dbf9f4d42a687910e854a685b9d
I20260619 16:46:19.469175 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12985949eebe494a97d3e6c8425a5414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:19.469202 6282 tablet_replica.cc:330] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: stopping tablet replica
I20260619 16:46:19.469547 6280 ts_tablet_manager.cc:1939] T e8651dbf9f4d42a687910e854a685b9d P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting consensus metadata
I20260619 16:46:19.469551 6282 raft_consensus.cc:2229] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:19.469950 6282 raft_consensus.cc:2258] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:19.470402 6011 tablet_service.cc:1467] Processing CreateTablet for tablet 12985949eebe494a97d3e6c8425a5414 (DEFAULT_TABLE table=test-table [id=a85b0b7ee4394c1baddb9e16592ab38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:19.470670 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12985949eebe494a97d3e6c8425a5414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:19.471577 6240 tablet_bootstrap.cc:492] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: Bootstrap starting.
I20260619 16:46:19.472409 6282 ts_tablet_manager.cc:1905] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:19.472522 6240 tablet_bootstrap.cc:654] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:19.473439 5721 catalog_manager.cc:4841] TS 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713): tablet e8651dbf9f4d42a687910e854a685b9d (table test-table [id=fe68da6ef6c345a9bf743c4a80683f4c]) successfully deleted
I20260619 16:46:19.473748 6242 tablet_bootstrap.cc:492] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: Bootstrap starting.
I20260619 16:46:19.474284 6151 tablet_service.cc:1467] Processing CreateTablet for tablet 12985949eebe494a97d3e6c8425a5414 (DEFAULT_TABLE table=test-table [id=a85b0b7ee4394c1baddb9e16592ab38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:19.474581 6151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12985949eebe494a97d3e6c8425a5414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:19.474690 6242 tablet_bootstrap.cc:654] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:19.476038 6282 ts_tablet_manager.cc:1918] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:19.476207 6282 log.cc:1196] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/e8651dbf9f4d42a687910e854a685b9d
I20260619 16:46:19.476289 6242 tablet_bootstrap.cc:492] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: No bootstrap required, opened a new log
I20260619 16:46:19.476399 6242 ts_tablet_manager.cc:1397] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:46:19.476743 6282 ts_tablet_manager.cc:1939] T e8651dbf9f4d42a687910e854a685b9d P f006ce00276543818e4bcc9f2aebada5: Deleting consensus metadata
I20260619 16:46:19.476847 6242 raft_consensus.cc:348] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.476972 6242 raft_consensus.cc:374] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:19.477012 6242 raft_consensus.cc:729] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6591d76ee4490aafb8eefb7ab198e1, State: Initialized, Role: FOLLOWER
I20260619 16:46:19.477147 6242 consensus_queue.cc:260] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.477432 6242 ts_tablet_manager.cc:1428] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:19.477748 6241 tablet_bootstrap.cc:492] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: Bootstrap starting.
I20260619 16:46:19.478667 5721 catalog_manager.cc:4841] TS f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911): tablet e8651dbf9f4d42a687910e854a685b9d (table test-table [id=fe68da6ef6c345a9bf743c4a80683f4c]) successfully deleted
I20260619 16:46:19.478878 6241 tablet_bootstrap.cc:654] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:19.468876 6281 log.cc:1196] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/e8651dbf9f4d42a687910e854a685b9d
I20260619 16:46:19.480320 6281 ts_tablet_manager.cc:1939] T e8651dbf9f4d42a687910e854a685b9d P 2dc59c5e80204b9bad858d983ecfa36f: Deleting consensus metadata
I20260619 16:46:19.480826 6240 tablet_bootstrap.cc:492] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: No bootstrap required, opened a new log
I20260619 16:46:19.480927 6240 ts_tablet_manager.cc:1397] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 16:46:19.481431 6240 raft_consensus.cc:348] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.481557 6240 raft_consensus.cc:374] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:19.481591 6240 raft_consensus.cc:729] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc59c5e80204b9bad858d983ecfa36f, State: Initialized, Role: FOLLOWER
I20260619 16:46:19.481698 6240 consensus_queue.cc:260] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.481994 6240 ts_tablet_manager.cc:1428] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:46:19.482304 6241 tablet_bootstrap.cc:492] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: No bootstrap required, opened a new log
I20260619 16:46:19.482396 6241 ts_tablet_manager.cc:1397] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 16:46:19.482559 5718 catalog_manager.cc:4841] TS 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457): tablet e8651dbf9f4d42a687910e854a685b9d (table test-table [id=fe68da6ef6c345a9bf743c4a80683f4c]) successfully deleted
I20260619 16:46:19.482985 6241 raft_consensus.cc:348] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.483116 6241 raft_consensus.cc:374] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:19.483153 6241 raft_consensus.cc:729] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Initialized, Role: FOLLOWER
I20260619 16:46:19.483289 6241 consensus_queue.cc:260] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.483628 6241 ts_tablet_manager.cc:1428] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:46:19.496644 6246 raft_consensus.cc:482] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:19.496819 6246 raft_consensus.cc:504] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.498008 6246 leader_election.cc:288] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713), f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911)
I20260619 16:46:19.503475 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.498308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:50523 (local address 127.3.233.130:34713)
0619 16:46:19.498466 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.498471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.498622 (+ 151us) server_negotiation.cc:407] Connection header received
0619 16:46:19.498804 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.498807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.498850 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.498893 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:19.499348 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.500227 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.502062 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.502722 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.502844 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.502914 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.503256 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.503336 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:46:19.503502 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.498238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34713 (local address 127.3.233.129:50523)
0619 16:46:19.498532 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.498548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.498695 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.499072 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.499074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.499091 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:19.499206 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.499211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.500370 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.500373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.501883 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.501907 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.502891 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.502899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.503109 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.503130 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.503225 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.503312 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":28,"thread_start_us":112,"threads_started":1}
I20260619 16:46:19.503569 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.498771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:53185 (local address 127.3.233.131:38911)
0619 16:46:19.498895 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.498902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.499136 (+ 234us) server_negotiation.cc:407] Connection header received
0619 16:46:19.499327 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.499330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.499384 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.499464 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:19.499751 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.500511 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.502174 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.502829 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.502981 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.503040 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.503365 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.503446 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:46:19.503966 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.498673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38911 (local address 127.3.233.129:53185)
0619 16:46:19.499047 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.499059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.499186 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.499486 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.499489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.499505 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:19.499627 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.499632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.500667 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.500671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.501994 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.502006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.503026 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.503040 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.503219 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.503242 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.503755 (+ 513us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.503831 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":22,"thread_start_us":107,"threads_started":1}
I20260619 16:46:19.504442 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12985949eebe494a97d3e6c8425a5414" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" is_pre_election: true
I20260619 16:46:19.504616 6031 raft_consensus.cc:2454] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 0.
I20260619 16:46:19.504580 6171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12985949eebe494a97d3e6c8425a5414" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f006ce00276543818e4bcc9f2aebada5" is_pre_election: true
I20260619 16:46:19.504802 6171 raft_consensus.cc:2454] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 0.
I20260619 16:46:19.505040 5825 leader_election.cc:302] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc59c5e80204b9bad858d983ecfa36f, 9f6591d76ee4490aafb8eefb7ab198e1; no voters:
I20260619 16:46:19.505357 6246 raft_consensus.cc:2781] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:19.505419 6246 raft_consensus.cc:482] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:19.505460 6246 raft_consensus.cc:3037] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.506994 6246 raft_consensus.cc:504] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.507404 6246 leader_election.cc:288] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 election: Requested vote from peers 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713), f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911)
I20260619 16:46:19.507822 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12985949eebe494a97d3e6c8425a5414" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
I20260619 16:46:19.507961 6031 raft_consensus.cc:3037] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.509095 6031 raft_consensus.cc:2454] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 1.
I20260619 16:46:19.509397 5825 leader_election.cc:302] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc59c5e80204b9bad858d983ecfa36f, 9f6591d76ee4490aafb8eefb7ab198e1; no voters:
I20260619 16:46:19.509449 6171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12985949eebe494a97d3e6c8425a5414" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f006ce00276543818e4bcc9f2aebada5"
I20260619 16:46:19.509572 6171 raft_consensus.cc:3037] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:19.509600 6246 raft_consensus.cc:2781] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:19.509845 6246 raft_consensus.cc:686] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 1 LEADER]: Becoming Leader. State: Replica: 2dc59c5e80204b9bad858d983ecfa36f, State: Running, Role: LEADER
I20260619 16:46:19.510130 6246 consensus_queue.cc:237] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:19.510713 6171 raft_consensus.cc:2454] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 1.
I20260619 16:46:19.512840 5733 catalog_manager.cc:5495] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129). New cstate: current_term: 1 leader_uuid: "2dc59c5e80204b9bad858d983ecfa36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:19.536724 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.532403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.3.233.129:42457)
0619 16:46:19.532672 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.532677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.532693 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:19.532850 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.532852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.532911 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.533040 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:19.533297 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.533876 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.534780 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.535135 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.535263 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.535541 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.536030 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.536181 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.536219 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.536277 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260619 16:46:19.548144 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.544558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:51307 (local address 127.3.233.131:38911)
0619 16:46:19.544695 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.544700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.544763 (+ 63us) server_negotiation.cc:407] Connection header received
0619 16:46:19.544880 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.544883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.544935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.545021 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:19.545224 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.545982 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.546982 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.547324 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.547422 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.547684 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.547770 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.547839 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.547950 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.548000 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:46:19.548162 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.544358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38911 (local address 127.3.233.129:51307)
0619 16:46:19.544684 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.544698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.544792 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.545030 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.545032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.545037 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:19.545135 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.545140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.546117 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.546120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.546840 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.546850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.547435 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.547438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.547516 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.547563 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.547869 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.547874 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.547930 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.548000 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
I20260619 16:46:19.555646 6031 raft_consensus.cc:1261] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Refusing update from remote peer 2dc59c5e80204b9bad858d983ecfa36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:19.555811 6171 raft_consensus.cc:1261] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Refusing update from remote peer 2dc59c5e80204b9bad858d983ecfa36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:19.556257 6288 consensus_queue.cc:1035] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:19.556465 6246 consensus_queue.cc:1035] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:19.558998 6266 mvcc.cc:205] Tried to move back new op lower bound from 7298611525855051776 to 7298611525676744704. Current Snapshot: MvccSnapshot[applied={T|T < 7298611525855051776}]
I20260619 16:46:19.575609 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.571300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42457 (local address 127.3.233.131:60081)
0619 16:46:19.571564 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:19.571581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:19.571670 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:19.572064 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:19.572067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:19.572078 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:19.572202 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.572210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.573213 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.573217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.574144 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:19.574165 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.574791 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.574795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:19.574887 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.574914 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.575290 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.575298 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:46:19.575364 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:46:19.575456 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 16:46:19.576069 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:19.571505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:60081 (local address 127.3.233.129:42457)
0619 16:46:19.571625 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:19.571630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:19.571644 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:19.571801 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:19.571804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:19.571859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:19.571929 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:19.572337 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.573047 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.574321 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:19.574676 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:19.574796 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:19.575075 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:19.575175 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:19.575248 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:19.575904 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 16:46:19.575962 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 16:46:19.576897 6266 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:46:21.139158 5733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:21.139305 5733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:21.141748 5733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Sending DeleteTablet for 3 replicas of tablet 12985949eebe494a97d3e6c8425a5414
I20260619 16:46:21.142264 5871 tablet_service.cc:1514] Processing DeleteTablet for tablet 12985949eebe494a97d3e6c8425a5414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:21 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:33764
I20260619 16:46:21.142486 6011 tablet_service.cc:1514] Processing DeleteTablet for tablet 12985949eebe494a97d3e6c8425a5414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:21 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50644
I20260619 16:46:21.142525 6151 tablet_service.cc:1514] Processing DeleteTablet for tablet 12985949eebe494a97d3e6c8425a5414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:21 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:34946
I20260619 16:46:21.142859 6322 tablet_replica.cc:330] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: stopping tablet replica
I20260619 16:46:21.142900 6323 tablet_replica.cc:330] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: stopping tablet replica
I20260619 16:46:21.143057 6323 raft_consensus.cc:2229] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:21.143100 6322 raft_consensus.cc:2229] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:21.143218 6323 raft_consensus.cc:2258] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:21.143378 6322 raft_consensus.cc:2258] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:21.144115 6322 ts_tablet_manager.cc:1905] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:21.146170 6321 tablet_replica.cc:330] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: stopping tablet replica
I20260619 16:46:21.146306 6321 raft_consensus.cc:2229] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:21.146382 6323 ts_tablet_manager.cc:1905] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:21.146634 6321 raft_consensus.cc:2258] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:21.147171 6322 ts_tablet_manager.cc:1918] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:46:21.147256 6322 log.cc:1196] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/12985949eebe494a97d3e6c8425a5414
I20260619 16:46:21.147307 6321 ts_tablet_manager.cc:1905] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:21.147508 6322 ts_tablet_manager.cc:1939] T 12985949eebe494a97d3e6c8425a5414 P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting consensus metadata
I20260619 16:46:21.148171 5721 catalog_manager.cc:4841] TS 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713): tablet 12985949eebe494a97d3e6c8425a5414 (table test-table [id=a85b0b7ee4394c1baddb9e16592ab38d]) successfully deleted
I20260619 16:46:21.149896 6321 ts_tablet_manager.cc:1918] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:46:21.149986 6321 log.cc:1196] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/12985949eebe494a97d3e6c8425a5414
I20260619 16:46:21.150298 6321 ts_tablet_manager.cc:1939] T 12985949eebe494a97d3e6c8425a5414 P 2dc59c5e80204b9bad858d983ecfa36f: Deleting consensus metadata
I20260619 16:46:21.151115 5718 catalog_manager.cc:4841] TS 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457): tablet 12985949eebe494a97d3e6c8425a5414 (table test-table [id=a85b0b7ee4394c1baddb9e16592ab38d]) successfully deleted
I20260619 16:46:21.151401 6323 ts_tablet_manager.cc:1918] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:46:21.151479 6323 log.cc:1196] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/12985949eebe494a97d3e6c8425a5414
I20260619 16:46:21.151757 6323 ts_tablet_manager.cc:1939] T 12985949eebe494a97d3e6c8425a5414 P f006ce00276543818e4bcc9f2aebada5: Deleting consensus metadata
I20260619 16:46:21.152421 5721 catalog_manager.cc:4841] TS f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911): tablet 12985949eebe494a97d3e6c8425a5414 (table test-table [id=a85b0b7ee4394c1baddb9e16592ab38d]) successfully deleted
Jun 19 16:46:21 dist-test-slave-cgwx krb5kdc[5689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:46:21 dist-test-slave-cgwx krb5kdc[5689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:21.176545 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:21.162573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.3.233.190:34895)
0619 16:46:21.162825 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:21.162831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:21.162846 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:21.163020 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:21.163022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:21.163065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:21.163156 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:21.163834 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.164388 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.165183 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.165403 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.165480 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:21.172987 (+ 7507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:21.172991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:21.172993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:21.173021 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:21.174627 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:21.175262 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:21.175265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:21.175266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:21.175347 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:21.175630 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:21.175633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:21.175635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:21.175831 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:21.175978 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:21.176213 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:46:21.176349 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":109,"threads_started":1}
I20260619 16:46:21.179349 5733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:21.179787 5733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:21.184535 5871 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9694b813694cf9986efac2ea2f540b (DEFAULT_TABLE table=test-table [id=e884cbb245e3465cbf4086415fe5e151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:21.184618 6011 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9694b813694cf9986efac2ea2f540b (DEFAULT_TABLE table=test-table [id=e884cbb245e3465cbf4086415fe5e151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:21.184710 6151 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9694b813694cf9986efac2ea2f540b (DEFAULT_TABLE table=test-table [id=e884cbb245e3465cbf4086415fe5e151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:21.184873 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9694b813694cf9986efac2ea2f540b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:21.184934 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9694b813694cf9986efac2ea2f540b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:21.184988 6151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9694b813694cf9986efac2ea2f540b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:21.187007 6332 tablet_bootstrap.cc:492] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: Bootstrap starting.
I20260619 16:46:21.187690 6333 tablet_bootstrap.cc:492] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: Bootstrap starting.
I20260619 16:46:21.187806 6334 tablet_bootstrap.cc:492] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: Bootstrap starting.
I20260619 16:46:21.188485 6332 tablet_bootstrap.cc:654] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:21.188557 6333 tablet_bootstrap.cc:654] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:21.188611 6334 tablet_bootstrap.cc:654] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:21.189610 6333 tablet_bootstrap.cc:492] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: No bootstrap required, opened a new log
I20260619 16:46:21.189723 6333 ts_tablet_manager.cc:1397] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:21.189791 6334 tablet_bootstrap.cc:492] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: No bootstrap required, opened a new log
I20260619 16:46:21.189872 6334 ts_tablet_manager.cc:1397] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:21.190143 6333 raft_consensus.cc:348] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.190269 6333 raft_consensus.cc:374] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:21.190318 6333 raft_consensus.cc:729] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Initialized, Role: FOLLOWER
I20260619 16:46:21.190310 6334 raft_consensus.cc:348] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.190428 6334 raft_consensus.cc:374] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:21.190467 6334 raft_consensus.cc:729] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc59c5e80204b9bad858d983ecfa36f, State: Initialized, Role: FOLLOWER
I20260619 16:46:21.190459 6333 consensus_queue.cc:260] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.190618 6334 consensus_queue.cc:260] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.190730 6333 ts_tablet_manager.cc:1428] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:21.190900 6334 ts_tablet_manager.cc:1428] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:21.191421 6332 tablet_bootstrap.cc:492] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: No bootstrap required, opened a new log
I20260619 16:46:21.191524 6332 ts_tablet_manager.cc:1397] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:46:21.191905 6332 raft_consensus.cc:348] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.192024 6332 raft_consensus.cc:374] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:21.192065 6332 raft_consensus.cc:729] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6591d76ee4490aafb8eefb7ab198e1, State: Initialized, Role: FOLLOWER
I20260619 16:46:21.192204 6332 consensus_queue.cc:260] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.192605 6332 ts_tablet_manager.cc:1428] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:21.390367 6299 raft_consensus.cc:482] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:21.390517 6299 raft_consensus.cc:504] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.391039 6299 leader_election.cc:288] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457), 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713)
I20260619 16:46:21.391438 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9694b813694cf9986efac2ea2f540b" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc59c5e80204b9bad858d983ecfa36f" is_pre_election: true
I20260619 16:46:21.391448 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9694b813694cf9986efac2ea2f540b" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" is_pre_election: true
I20260619 16:46:21.391597 5891 raft_consensus.cc:2454] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 0.
I20260619 16:46:21.391597 6031 raft_consensus.cc:2454] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 0.
I20260619 16:46:21.391943 6102 leader_election.cc:302] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dc59c5e80204b9bad858d983ecfa36f, 9f6591d76ee4490aafb8eefb7ab198e1, f006ce00276543818e4bcc9f2aebada5; no voters:
I20260619 16:46:21.392269 6299 raft_consensus.cc:2781] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:21.392333 6299 raft_consensus.cc:482] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:21.392361 6299 raft_consensus.cc:3037] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:21.393440 6299 raft_consensus.cc:504] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.393800 6299 leader_election.cc:288] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 election: Requested vote from peers 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457), 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713)
I20260619 16:46:21.394227 5891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9694b813694cf9986efac2ea2f540b" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc59c5e80204b9bad858d983ecfa36f"
I20260619 16:46:21.394227 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9694b813694cf9986efac2ea2f540b" candidate_uuid: "f006ce00276543818e4bcc9f2aebada5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
I20260619 16:46:21.394369 5891 raft_consensus.cc:3037] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:21.394369 6031 raft_consensus.cc:3037] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:21.395586 5891 raft_consensus.cc:2454] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 1.
I20260619 16:46:21.395593 6031 raft_consensus.cc:2454] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f006ce00276543818e4bcc9f2aebada5 in term 1.
I20260619 16:46:21.395889 6105 leader_election.cc:302] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6591d76ee4490aafb8eefb7ab198e1, f006ce00276543818e4bcc9f2aebada5; no voters:
I20260619 16:46:21.396093 6299 raft_consensus.cc:2781] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:21.396201 6299 raft_consensus.cc:686] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 1 LEADER]: Becoming Leader. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Running, Role: LEADER
I20260619 16:46:21.396342 6299 consensus_queue.cc:237] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.398155 5732 catalog_manager.cc:5495] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f006ce00276543818e4bcc9f2aebada5 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "f006ce00276543818e4bcc9f2aebada5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } health_report { overall_health: HEALTHY } } }
I20260619 16:46:21.435047 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:21.431576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.3.233.131:38911)
0619 16:46:21.431835 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:21.431841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:21.431853 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:46:21.431890 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:21.431892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:21.431931 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:21.431999 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:21.432208 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.432779 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.433802 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.434028 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.434125 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:21.434426 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:21.434554 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:21.434623 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:21.434767 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 16:46:21.434824 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":123,"threads_started":1}
I20260619 16:46:21.441500 6031 raft_consensus.cc:1261] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Refusing update from remote peer f006ce00276543818e4bcc9f2aebada5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:21.441710 5891 raft_consensus.cc:1261] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Refusing update from remote peer f006ce00276543818e4bcc9f2aebada5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:21.441886 6307 consensus_queue.cc:1035] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:21.442052 6302 consensus_queue.cc:1035] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:21.471997 5732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:21.472168 5732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:21.474363 5732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Sending DeleteTablet for 3 replicas of tablet 1f9694b813694cf9986efac2ea2f540b
I20260619 16:46:21.474884 5871 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9694b813694cf9986efac2ea2f540b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:21 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:33764
I20260619 16:46:21.474999 6150 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9694b813694cf9986efac2ea2f540b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:21 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:34946
I20260619 16:46:21.475126 6321 tablet_replica.cc:330] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: stopping tablet replica
I20260619 16:46:21.475118 6011 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9694b813694cf9986efac2ea2f540b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:21 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50644
I20260619 16:46:21.475247 6321 raft_consensus.cc:2229] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:21.475299 6322 tablet_replica.cc:330] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: stopping tablet replica
I20260619 16:46:21.475307 6323 tablet_replica.cc:330] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: stopping tablet replica
I20260619 16:46:21.475356 6321 raft_consensus.cc:2258] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:21.475409 6323 raft_consensus.cc:2229] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:21.475407 6322 raft_consensus.cc:2229] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:21.475507 6322 raft_consensus.cc:2258] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:21.475625 6323 raft_consensus.cc:2258] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:21.475701 5732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:21.476114 5732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:21.476274 6321 ts_tablet_manager.cc:1905] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:21.476392 6323 ts_tablet_manager.cc:1905] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:21.476778 6322 ts_tablet_manager.cc:1905] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:21.478561 6323 ts_tablet_manager.cc:1918] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:21.478617 6323 log.cc:1196] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/1f9694b813694cf9986efac2ea2f540b
I20260619 16:46:21.478801 6323 ts_tablet_manager.cc:1939] T 1f9694b813694cf9986efac2ea2f540b P f006ce00276543818e4bcc9f2aebada5: Deleting consensus metadata
I20260619 16:46:21.479007 6321 ts_tablet_manager.cc:1918] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:21.479099 6321 log.cc:1196] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/1f9694b813694cf9986efac2ea2f540b
I20260619 16:46:21.479315 6322 ts_tablet_manager.cc:1918] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:21.479360 6321 ts_tablet_manager.cc:1939] T 1f9694b813694cf9986efac2ea2f540b P 2dc59c5e80204b9bad858d983ecfa36f: Deleting consensus metadata
I20260619 16:46:21.479390 6322 log.cc:1196] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/1f9694b813694cf9986efac2ea2f540b
I20260619 16:46:21.479564 5721 catalog_manager.cc:4841] TS f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911): tablet 1f9694b813694cf9986efac2ea2f540b (table test-table [id=e884cbb245e3465cbf4086415fe5e151]) successfully deleted
I20260619 16:46:21.479632 6322 ts_tablet_manager.cc:1939] T 1f9694b813694cf9986efac2ea2f540b P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting consensus metadata
I20260619 16:46:21.480085 5718 catalog_manager.cc:4841] TS 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457): tablet 1f9694b813694cf9986efac2ea2f540b (table test-table [id=e884cbb245e3465cbf4086415fe5e151]) successfully deleted
I20260619 16:46:21.480934 5721 catalog_manager.cc:4841] TS 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713): tablet 1f9694b813694cf9986efac2ea2f540b (table test-table [id=e884cbb245e3465cbf4086415fe5e151]) successfully deleted
I20260619 16:46:21.483403 5871 tablet_service.cc:1467] Processing CreateTablet for tablet ff262ebe96f84779b0f29525713fb59a (DEFAULT_TABLE table=test-table [id=f592742ae61c47e5a072b5d9850e090d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:21.483563 6011 tablet_service.cc:1467] Processing CreateTablet for tablet ff262ebe96f84779b0f29525713fb59a (DEFAULT_TABLE table=test-table [id=f592742ae61c47e5a072b5d9850e090d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:21.483721 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff262ebe96f84779b0f29525713fb59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:21.483851 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff262ebe96f84779b0f29525713fb59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:21.483934 6150 tablet_service.cc:1467] Processing CreateTablet for tablet ff262ebe96f84779b0f29525713fb59a (DEFAULT_TABLE table=test-table [id=f592742ae61c47e5a072b5d9850e090d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:21.484215 6150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff262ebe96f84779b0f29525713fb59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:21.485495 6334 tablet_bootstrap.cc:492] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f: Bootstrap starting.
I20260619 16:46:21.486261 6332 tablet_bootstrap.cc:492] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: Bootstrap starting.
I20260619 16:46:21.486445 6334 tablet_bootstrap.cc:654] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:21.486821 6333 tablet_bootstrap.cc:492] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: Bootstrap starting.
I20260619 16:46:21.487241 6332 tablet_bootstrap.cc:654] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:21.487819 6333 tablet_bootstrap.cc:654] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:21.487936 6334 tablet_bootstrap.cc:492] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f: No bootstrap required, opened a new log
I20260619 16:46:21.488024 6334 ts_tablet_manager.cc:1397] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 16:46:21.488415 6332 tablet_bootstrap.cc:492] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: No bootstrap required, opened a new log
I20260619 16:46:21.488495 6332 ts_tablet_manager.cc:1397] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:21.488457 6334 raft_consensus.cc:348] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.488561 6334 raft_consensus.cc:374] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:21.488615 6334 raft_consensus.cc:729] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc59c5e80204b9bad858d983ecfa36f, State: Initialized, Role: FOLLOWER
I20260619 16:46:21.488736 6334 consensus_queue.cc:260] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.488914 6332 raft_consensus.cc:348] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.489009 6334 ts_tablet_manager.cc:1428] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:46:21.489030 6332 raft_consensus.cc:374] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:21.489066 6333 tablet_bootstrap.cc:492] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: No bootstrap required, opened a new log
I20260619 16:46:21.489060 6332 raft_consensus.cc:729] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6591d76ee4490aafb8eefb7ab198e1, State: Initialized, Role: FOLLOWER
I20260619 16:46:21.489133 6333 ts_tablet_manager.cc:1397] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:21.489194 6332 consensus_queue.cc:260] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.489491 6332 ts_tablet_manager.cc:1428] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:21.489506 6333 raft_consensus.cc:348] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.489619 6333 raft_consensus.cc:374] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:21.489652 6333 raft_consensus.cc:729] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f006ce00276543818e4bcc9f2aebada5, State: Initialized, Role: FOLLOWER
I20260619 16:46:21.489802 6333 consensus_queue.cc:260] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.490135 6333 ts_tablet_manager.cc:1428] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:21.650079 6312 raft_consensus.cc:482] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:21.650213 6312 raft_consensus.cc:504] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.650710 6312 leader_election.cc:288] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713), f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911)
I20260619 16:46:21.651209 6171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff262ebe96f84779b0f29525713fb59a" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f006ce00276543818e4bcc9f2aebada5" is_pre_election: true
I20260619 16:46:21.651233 6030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff262ebe96f84779b0f29525713fb59a" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" is_pre_election: true
I20260619 16:46:21.651373 6171 raft_consensus.cc:2454] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 0.
I20260619 16:46:21.651373 6030 raft_consensus.cc:2454] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 0.
I20260619 16:46:21.651675 5825 leader_election.cc:302] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc59c5e80204b9bad858d983ecfa36f, f006ce00276543818e4bcc9f2aebada5; no voters:
I20260619 16:46:21.651834 6312 raft_consensus.cc:2781] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:21.651892 6312 raft_consensus.cc:482] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:21.651927 6312 raft_consensus.cc:3037] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:21.652900 6312 raft_consensus.cc:504] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.653213 6312 leader_election.cc:288] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 election: Requested vote from peers 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713), f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911)
I20260619 16:46:21.653564 6171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff262ebe96f84779b0f29525713fb59a" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f006ce00276543818e4bcc9f2aebada5"
I20260619 16:46:21.653587 6030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff262ebe96f84779b0f29525713fb59a" candidate_uuid: "2dc59c5e80204b9bad858d983ecfa36f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6591d76ee4490aafb8eefb7ab198e1"
I20260619 16:46:21.653685 6030 raft_consensus.cc:3037] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:21.653712 6171 raft_consensus.cc:3037] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:21.654681 6030 raft_consensus.cc:2454] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 1.
I20260619 16:46:21.654879 6171 raft_consensus.cc:2454] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dc59c5e80204b9bad858d983ecfa36f in term 1.
I20260619 16:46:21.655015 5825 leader_election.cc:302] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dc59c5e80204b9bad858d983ecfa36f, 9f6591d76ee4490aafb8eefb7ab198e1; no voters:
I20260619 16:46:21.655220 6312 raft_consensus.cc:2781] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:21.655404 6312 raft_consensus.cc:686] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [term 1 LEADER]: Becoming Leader. State: Replica: 2dc59c5e80204b9bad858d983ecfa36f, State: Running, Role: LEADER
I20260619 16:46:21.655572 6312 consensus_queue.cc:237] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } }
I20260619 16:46:21.657071 5734 catalog_manager.cc:5495] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129). New cstate: current_term: 1 leader_uuid: "2dc59c5e80204b9bad858d983ecfa36f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dc59c5e80204b9bad858d983ecfa36f" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:21.688275 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:21.684662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.3.233.129:42457)
0619 16:46:21.684875 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:21.684890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:21.684903 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:21.684945 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:21.684947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:21.684992 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:21.685082 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:21.685341 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.686153 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.687068 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:21.687367 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:21.687474 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:21.687771 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:21.687856 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:21.687966 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:21.688064 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 16:46:21.688114 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260619 16:46:21.695024 6030 raft_consensus.cc:1261] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Refusing update from remote peer 2dc59c5e80204b9bad858d983ecfa36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:21.695024 6171 raft_consensus.cc:1261] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Refusing update from remote peer 2dc59c5e80204b9bad858d983ecfa36f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:21.695489 6312 consensus_queue.cc:1035] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6591d76ee4490aafb8eefb7ab198e1" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:21.695744 6349 consensus_queue.cc:1035] T ff262ebe96f84779b0f29525713fb59a P 2dc59c5e80204b9bad858d983ecfa36f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f006ce00276543818e4bcc9f2aebada5" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:46:21.708003 6303 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:46:23.256731 5731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:23.256873 5731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:23.259258 5731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ad6865ac6ed496c80e1d99a4daa5394: Sending DeleteTablet for 3 replicas of tablet ff262ebe96f84779b0f29525713fb59a
I20260619 16:46:23.259855 5871 tablet_service.cc:1514] Processing DeleteTablet for tablet ff262ebe96f84779b0f29525713fb59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:23 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:33764
I20260619 16:46:23.259918 6011 tablet_service.cc:1514] Processing DeleteTablet for tablet ff262ebe96f84779b0f29525713fb59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:23 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50644
I20260619 16:46:23.259899 6150 tablet_service.cc:1514] Processing DeleteTablet for tablet ff262ebe96f84779b0f29525713fb59a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:23 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:34946
I20260619 16:46:23.260216 6379 tablet_replica.cc:330] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: stopping tablet replica
I20260619 16:46:23.260295 6381 tablet_replica.cc:330] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: stopping tablet replica
I20260619 16:46:23.260407 6379 raft_consensus.cc:2229] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:23.260417 6381 raft_consensus.cc:2229] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:23.260521 6381 raft_consensus.cc:2258] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:23.260541 6379 raft_consensus.cc:2258] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:23.260604 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 5806
I20260619 16:46:23.261693 6381 ts_tablet_manager.cc:1905] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:23.261881 6379 ts_tablet_manager.cc:1905] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:23.263908 6379 ts_tablet_manager.cc:1918] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:46:23.263995 6379 log.cc:1196] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/ff262ebe96f84779b0f29525713fb59a
I20260619 16:46:23.264240 6379 ts_tablet_manager.cc:1939] T ff262ebe96f84779b0f29525713fb59a P 9f6591d76ee4490aafb8eefb7ab198e1: Deleting consensus metadata
I20260619 16:46:23.264783 6381 ts_tablet_manager.cc:1918] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:46:23.264866 6381 log.cc:1196] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/ff262ebe96f84779b0f29525713fb59a
I20260619 16:46:23.264840 5721 catalog_manager.cc:4841] TS 9f6591d76ee4490aafb8eefb7ab198e1 (127.3.233.130:34713): tablet ff262ebe96f84779b0f29525713fb59a (table test-table [id=f592742ae61c47e5a072b5d9850e090d]) successfully deleted
I20260619 16:46:23.265136 6381 ts_tablet_manager.cc:1939] T ff262ebe96f84779b0f29525713fb59a P f006ce00276543818e4bcc9f2aebada5: Deleting consensus metadata
I20260619 16:46:23.265887 5721 catalog_manager.cc:4841] TS f006ce00276543818e4bcc9f2aebada5 (127.3.233.131:38911): tablet ff262ebe96f84779b0f29525713fb59a (table test-table [id=f592742ae61c47e5a072b5d9850e090d]) successfully deleted
W20260619 16:46:23.270130 5718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:42457 (error 108)
I20260619 16:46:23.270449 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 5946
I20260619 16:46:23.273685 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:23.273286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42457 (local address 127.0.0.1:46648)
0619 16:46:23.273520 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:23.273598 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:42457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
W20260619 16:46:23.273809 5718 catalog_manager.cc:4584] TS 2dc59c5e80204b9bad858d983ecfa36f (127.3.233.129:42457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff262ebe96f84779b0f29525713fb59a: Network error: Client connection negotiation failed: client connection to 127.3.233.129:42457: connect: Connection refused (error 111)
I20260619 16:46:23.276535 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 6086
I20260619 16:46:23.284394 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 5702
2026-06-19T16:46:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6494 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:23 dist-test-slave-cgwx krb5kdc[6386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:23 dist-test-slave-cgwx krb5kdc[6386](info): set up 2 sockets
Jun 19 16:46:23 dist-test-slave-cgwx krb5kdc[6386](info): commencing operation
krb5kdc: starting...
W20260619 16:46:24.355428 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:24 dist-test-slave-cgwx krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:24.475941 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:44341
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:24.586043 6399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:24.586352 6399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:24.586421 6399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:24.590422 6399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:24.590510 6399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:24.590551 6399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:24.590603 6399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:24.590655 6399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:24.596990 6399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:44341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:24.598201 6399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:24.599319 6399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:24.608366 6407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.608425 6404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.608431 6405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:24.608587 6399 server_base.cc:1034] running on GCE node
I20260619 16:46:24.609321 6399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:24.610553 6399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:24.611737 6399 hybrid_clock.cc:648] HybridClock initialized: now 1781887584611715 us; error 39 us; skew 500 ppm
Jun 19 16:46:24 dist-test-slave-cgwx krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:24.614768 6399 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:24.615736 6399 webserver.cc:460] Webserver started at http://127.3.233.190:43293/ using document root <none> and password file <none>
I20260619 16:46:24.616266 6399 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:24.616326 6399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:24.616492 6399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:24.618489 6399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "5eda3fa1e1974459bdf41af7e886a020"
format_stamp: "Formatted at 2026-06-19 16:46:24 on dist-test-slave-cgwx"
server_key: "8930d4c21ee7a1deda4e18561756b2ee"
server_key_iv: "9c10d38f7622d7543e5477c0f8d4492e"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.619071 6399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "5eda3fa1e1974459bdf41af7e886a020"
format_stamp: "Formatted at 2026-06-19 16:46:24 on dist-test-slave-cgwx"
server_key: "8930d4c21ee7a1deda4e18561756b2ee"
server_key_iv: "9c10d38f7622d7543e5477c0f8d4492e"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.622865 6399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:46:24.625358 6414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.626740 6399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:46:24.626848 6399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "5eda3fa1e1974459bdf41af7e886a020"
format_stamp: "Formatted at 2026-06-19 16:46:24 on dist-test-slave-cgwx"
server_key: "8930d4c21ee7a1deda4e18561756b2ee"
server_key_iv: "9c10d38f7622d7543e5477c0f8d4492e"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.627002 6399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:24.656301 6399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:24.659533 6399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:24.659698 6399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:24.666402 6399 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:44341
I20260619 16:46:24.666481 6466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:44341 every 8 connection(s)
I20260619 16:46:24.667436 6399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:24.670258 6467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:24.672674 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 6399
I20260619 16:46:24.672765 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:24.673035 4006 external_mini_cluster.cc:1428] Setting key a31afee834cd8bf4f064327c3d7c98c4
I20260619 16:46:24.676286 6467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020: Bootstrap starting.
I20260619 16:46:24.678836 6467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:24.679703 6467 log.cc:824] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:24.681753 6467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020: No bootstrap required, opened a new log
Jun 19 16:46:24 dist-test-slave-cgwx krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:24.685936 6467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eda3fa1e1974459bdf41af7e886a020" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44341 } }
I20260619 16:46:24.686240 6467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:24.686302 6467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eda3fa1e1974459bdf41af7e886a020, State: Initialized, Role: FOLLOWER
I20260619 16:46:24.686831 6467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eda3fa1e1974459bdf41af7e886a020" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44341 } }
I20260619 16:46:24.687005 6467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:24.687074 6467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:24.687175 6467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:24.688344 6467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eda3fa1e1974459bdf41af7e886a020" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44341 } }
I20260619 16:46:24.688685 6467 leader_election.cc:302] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eda3fa1e1974459bdf41af7e886a020; no voters:
I20260619 16:46:24.688925 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:24.674516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.3.233.190:44341)
0619 16:46:24.674912 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:24.674976 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:24.675023 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:46:24.675639 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:24.675656 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:24.675999 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:24.676284 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:24.677199 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.678183 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.679379 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.679983 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.680155 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:24.682822 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:24.682829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:24.682845 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:24.682876 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:24.685528 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:24.686045 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:24.686050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:24.686056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:24.686192 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:24.686570 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:24.686574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:24.686575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:24.687117 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:24.687335 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:24.687566 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 16:46:24.687867 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":139,"threads_started":1}
I20260619 16:46:24.689054 6467 leader_election.cc:288] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:24.689507 6473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:24.689898 6473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [term 1 LEADER]: Becoming Leader. State: Replica: 5eda3fa1e1974459bdf41af7e886a020, State: Running, Role: LEADER
I20260619 16:46:24.690325 6467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:24.690827 6473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eda3fa1e1974459bdf41af7e886a020" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44341 } }
I20260619 16:46:24.692981 6472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eda3fa1e1974459bdf41af7e886a020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eda3fa1e1974459bdf41af7e886a020" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44341 } } }
I20260619 16:46:24.693073 6472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:24.692801 6474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eda3fa1e1974459bdf41af7e886a020. Latest consensus state: current_term: 1 leader_uuid: "5eda3fa1e1974459bdf41af7e886a020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eda3fa1e1974459bdf41af7e886a020" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44341 } } }
I20260619 16:46:24.693351 6474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020 [sys.catalog]: This master's current role is: LEADER
W20260619 16:46:24.696072 6486 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:24.696166 6486 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:24.696255 6487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:24.697014 6487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:24.702531 6487 catalog_manager.cc:1347] Generated new cluster ID: dac3a21251ca4684b02cb51a1b7f6981
I20260619 16:46:24.702601 6487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:24.715852 6487 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:24.716876 6487 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:24.729537 6487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eda3fa1e1974459bdf41af7e886a020: Generated new TSK 0
I20260619 16:46:24.730337 6487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:24.761446 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44341
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:24.884116 6493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:24.884439 6493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:24.884533 6493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:24.889145 6493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:24.889210 6493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:24.889307 6493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:24.895491 6493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:24.896637 6493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:24.897809 6493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:24.907995 6499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.907996 6498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:24.907995 6501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:24.908470 6493 server_base.cc:1034] running on GCE node
I20260619 16:46:24.908892 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:24.909487 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:24.910720 6493 hybrid_clock.cc:648] HybridClock initialized: now 1781887584910726 us; error 49 us; skew 500 ppm
Jun 19 16:46:24 dist-test-slave-cgwx krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:24.913609 6493 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:24.914454 6493 webserver.cc:460] Webserver started at http://127.3.233.129:34669/ using document root <none> and password file <none>
I20260619 16:46:24.915059 6493 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:24.915156 6493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:24.915326 6493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:24.917482 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "8bcbe5021f2e4a548a72e6c275600a8f"
format_stamp: "Formatted at 2026-06-19 16:46:24 on dist-test-slave-cgwx"
server_key: "9201a84d792352eab7c6716a95ffe1a6"
server_key_iv: "d8e1af3f8c3b377956af0cd433833ed7"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.918046 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "8bcbe5021f2e4a548a72e6c275600a8f"
format_stamp: "Formatted at 2026-06-19 16:46:24 on dist-test-slave-cgwx"
server_key: "9201a84d792352eab7c6716a95ffe1a6"
server_key_iv: "d8e1af3f8c3b377956af0cd433833ed7"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.922142 6493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:46:24.924885 6508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.926606 6493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:24.926717 6493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "8bcbe5021f2e4a548a72e6c275600a8f"
format_stamp: "Formatted at 2026-06-19 16:46:24 on dist-test-slave-cgwx"
server_key: "9201a84d792352eab7c6716a95ffe1a6"
server_key_iv: "d8e1af3f8c3b377956af0cd433833ed7"
server_key_version: "encryptionkey@0"
I20260619 16:46:24.926853 6493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:24.945904 6493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:24.949452 6493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:24.949618 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:24.950183 6493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:24.951012 6493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:24.951061 6493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.951100 6493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:24.951115 6493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:24.963469 6493 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:35571
I20260619 16:46:24.963487 6621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:35571 every 8 connection(s)
I20260619 16:46:24.964359 6493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:24.968408 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 6493
I20260619 16:46:24.968547 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:24.968827 4006 external_mini_cluster.cc:1428] Setting key b82b8267530978c09dec5b40bfd5cb8c
Jun 19 16:46:24 dist-test-slave-cgwx krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887584, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:24.979946 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:24.966229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:35061 (local address 127.3.233.190:44341)
0619 16:46:24.966380 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:24.966384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:24.967330 (+ 946us) server_negotiation.cc:407] Connection header received
0619 16:46:24.968795 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:24.968798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:24.968859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:24.968958 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:24.970446 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.971060 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.972588 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.972993 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.973131 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:24.975765 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:24.975772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:24.975777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:24.975812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:24.977562 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:24.978113 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:24.978117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:24.978119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:24.978223 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:24.978623 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:24.978626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:24.978629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:24.978882 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:24.979020 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:24.979695 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 16:46:24.979817 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:24.981084 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:24.966637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:44341 (local address 127.3.233.129:35061)
0619 16:46:24.967169 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:24.967216 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:24.968508 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:24.969218 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:24.969229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:24.969649 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:24.970278 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:24.970295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.971186 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.971192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:24.972457 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:24.972467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:24.973165 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:24.973168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:24.973329 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:24.973949 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:24.973966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:24.975577 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:24.977707 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:24.977714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:24.977728 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:24.978002 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:24.978321 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:24.978324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:24.978325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:24.978466 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:24.979072 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:24.979078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:24.979405 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 16:46:24.979647 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 16:46:24.979974 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":99,"threads_started":1}
I20260619 16:46:24.982579 6622 heartbeater.cc:347] Connected to a master server at 127.3.233.190:44341
I20260619 16:46:24.982973 6622 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:24.983531 6622 heartbeater.cc:511] Master 127.3.233.190:44341 requested a full tablet report, sending...
I20260619 16:46:24.985136 6429 ts_manager.cc:194] Registered new tserver with Master: 8bcbe5021f2e4a548a72e6c275600a8f (127.3.233.129:35571)
I20260619 16:46:24.986377 6429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:35061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:24.997859 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44341
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:25.109968 6627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:25.110268 6627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:25.110314 6627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:25.114418 6627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:25.114476 6627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:25.114538 6627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:25.119757 6627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:25.120774 6627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:25.121732 6627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:25.132266 6633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:25.132368 6627 server_base.cc:1034] running on GCE node
W20260619 16:46:25.132513 6632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:25.132282 6635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:25.132961 6627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:25.133656 6627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:25.134893 6627 hybrid_clock.cc:648] HybridClock initialized: now 1781887585134871 us; error 37 us; skew 500 ppm
Jun 19 16:46:25 dist-test-slave-cgwx krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:25.137714 6627 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:25.138564 6627 webserver.cc:460] Webserver started at http://127.3.233.130:39525/ using document root <none> and password file <none>
I20260619 16:46:25.139202 6627 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:25.139292 6627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:25.139511 6627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:25.141518 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "957a3d4d902e4ab1845b4e3abc6cb2b8"
format_stamp: "Formatted at 2026-06-19 16:46:25 on dist-test-slave-cgwx"
server_key: "c0cd25dc6caf476a9ce4921b23e1fa40"
server_key_iv: "47ee8abebf9f489aeade9e8778e784f5"
server_key_version: "encryptionkey@0"
I20260619 16:46:25.142062 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "957a3d4d902e4ab1845b4e3abc6cb2b8"
format_stamp: "Formatted at 2026-06-19 16:46:25 on dist-test-slave-cgwx"
server_key: "c0cd25dc6caf476a9ce4921b23e1fa40"
server_key_iv: "47ee8abebf9f489aeade9e8778e784f5"
server_key_version: "encryptionkey@0"
I20260619 16:46:25.145705 6627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 16:46:25.148084 6642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.149304 6627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:25.149432 6627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "957a3d4d902e4ab1845b4e3abc6cb2b8"
format_stamp: "Formatted at 2026-06-19 16:46:25 on dist-test-slave-cgwx"
server_key: "c0cd25dc6caf476a9ce4921b23e1fa40"
server_key_iv: "47ee8abebf9f489aeade9e8778e784f5"
server_key_version: "encryptionkey@0"
I20260619 16:46:25.149538 6627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:25.177091 6627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:25.180220 6627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:25.180411 6627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:25.181008 6627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:25.181814 6627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:25.181881 6627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.181947 6627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:25.181988 6627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.191349 6627 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:40835
I20260619 16:46:25.191407 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:40835 every 8 connection(s)
I20260619 16:46:25.192407 6627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:25.194447 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 6627
I20260619 16:46:25.194548 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:25.194836 4006 external_mini_cluster.cc:1428] Setting key eae70ff646856d40b6ceb83109cbd06a
Jun 19 16:46:25 dist-test-slave-cgwx krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:46:25.206545 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.194396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:36733 (local address 127.3.233.190:44341)
0619 16:46:25.194551 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.194558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.195517 (+ 959us) server_negotiation.cc:407] Connection header received
0619 16:46:25.196917 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.196923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.196995 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.197092 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.198288 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.198813 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.199982 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.200223 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.200333 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.202887 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:25.202891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:25.202893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:25.202960 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:25.204416 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.204958 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.204963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.204964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.205043 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.205390 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.205395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.205397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.205614 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:25.205727 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.206252 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.206381 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:46:25.207473 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.194838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:44341 (local address 127.3.233.130:36733)
0619 16:46:25.195362 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.195417 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.196702 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.197253 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.197260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.197617 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:25.198138 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.198151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.198958 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.198962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.199855 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.199863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.200373 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.200378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.200544 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.201229 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:25.201247 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:25.202720 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:25.204533 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.204539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.204556 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.204857 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.205139 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.205142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.205144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.205296 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.205754 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:25.205760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:25.206002 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.206205 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.206538 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":135,"threads_started":1}
I20260619 16:46:25.208938 6756 heartbeater.cc:347] Connected to a master server at 127.3.233.190:44341
I20260619 16:46:25.209235 6756 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:25.209872 6756 heartbeater.cc:511] Master 127.3.233.190:44341 requested a full tablet report, sending...
I20260619 16:46:25.211158 6429 ts_manager.cc:194] Registered new tserver with Master: 957a3d4d902e4ab1845b4e3abc6cb2b8 (127.3.233.130:40835)
I20260619 16:46:25.212111 6429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:36733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:25.224174 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44341
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:25.334507 6761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:25.334849 6761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:25.334945 6761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:25.339191 6761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:25.339279 6761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:25.339370 6761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:25.344791 6761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:25.345950 6761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:25.347076 6761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:25.357173 6769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:25.357218 6767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:25.357410 6761 server_base.cc:1034] running on GCE node
W20260619 16:46:25.357172 6766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:25.357900 6761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:25.358623 6761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:25.359812 6761 hybrid_clock.cc:648] HybridClock initialized: now 1781887585359765 us; error 59 us; skew 500 ppm
Jun 19 16:46:25 dist-test-slave-cgwx krb5kdc[6386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:25.362581 6761 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:25.363571 6761 webserver.cc:460] Webserver started at http://127.3.233.131:39593/ using document root <none> and password file <none>
I20260619 16:46:25.364159 6761 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:25.364250 6761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:25.364480 6761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:25.366544 6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "25427d44e5e54dd4a2a53aa67ba92190"
format_stamp: "Formatted at 2026-06-19 16:46:25 on dist-test-slave-cgwx"
server_key: "b998ce0c6a2cfd79076fc3ed63099a24"
server_key_iv: "d788f67635620ee0a6c488715d6fc671"
server_key_version: "encryptionkey@0"
I20260619 16:46:25.367180 6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "25427d44e5e54dd4a2a53aa67ba92190"
format_stamp: "Formatted at 2026-06-19 16:46:25 on dist-test-slave-cgwx"
server_key: "b998ce0c6a2cfd79076fc3ed63099a24"
server_key_iv: "d788f67635620ee0a6c488715d6fc671"
server_key_version: "encryptionkey@0"
I20260619 16:46:25.370860 6761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:25.373451 6776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.374759 6761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:46:25.374903 6761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "25427d44e5e54dd4a2a53aa67ba92190"
format_stamp: "Formatted at 2026-06-19 16:46:25 on dist-test-slave-cgwx"
server_key: "b998ce0c6a2cfd79076fc3ed63099a24"
server_key_iv: "d788f67635620ee0a6c488715d6fc671"
server_key_version: "encryptionkey@0"
I20260619 16:46:25.375067 6761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:25.386444 6761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:25.389370 6761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:25.389562 6761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:25.390131 6761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:25.391047 6761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:25.391112 6761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.391181 6761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:25.391225 6761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:25.400141 6761 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:42237
I20260619 16:46:25.400173 6889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:42237 every 8 connection(s)
I20260619 16:46:25.401005 6761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:25.410326 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 6761
I20260619 16:46:25.410454 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
Jun 19 16:46:25 dist-test-slave-cgwx krb5kdc[6386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:25.410888 4006 external_mini_cluster.cc:1428] Setting key 93b2e4264006d7532d45e9c74923b00e
I20260619 16:46:25.415136 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.402838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:39025 (local address 127.3.233.190:44341)
0619 16:46:25.402998 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.403002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.403829 (+ 827us) server_negotiation.cc:407] Connection header received
0619 16:46:25.405183 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.405186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.405240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.405320 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.406441 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.407045 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.408157 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.408390 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.408496 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.411399 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:25.411404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:25.411410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:25.411442 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:25.413162 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.413658 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.413663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.413665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.413742 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:25.414065 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:25.414071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:25.414073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:25.414254 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:25.414356 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:25.414837 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 16:46:25.414982 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:25.415932 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:25.403255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:44341 (local address 127.3.233.131:39025)
0619 16:46:25.403679 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:25.403722 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:25.404959 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:25.405482 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:25.405489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:25.405829 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:25.406297 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.406310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.407164 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.407168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.408026 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:25.408035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:25.408536 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:25.408540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:25.408707 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:25.409361 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:25.409376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:25.411211 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:25.413268 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.413275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.413291 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.413555 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.413832 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:25.413836 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:25.413838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:25.413974 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:25.414352 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:25.414359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:25.414603 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 16:46:25.414790 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 16:46:25.415149 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260619 16:46:25.417119 6890 heartbeater.cc:347] Connected to a master server at 127.3.233.190:44341
I20260619 16:46:25.417371 6890 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:25.417910 6890 heartbeater.cc:511] Master 127.3.233.190:44341 requested a full tablet report, sending...
I20260619 16:46:25.418915 6429 ts_manager.cc:194] Registered new tserver with Master: 25427d44e5e54dd4a2a53aa67ba92190 (127.3.233.131:42237)
I20260619 16:46:25.419552 6429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:39025
I20260619 16:46:25.424943 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:46:25.438133 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:25.436620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.3.233.190:44341)
0619 16:46:25.436778 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:25.436782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:25.436935 (+ 153us) server_negotiation.cc:407] Connection header received
0619 16:46:25.437047 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:25.437051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:25.437095 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:25.437205 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:25.437937 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47190: BlockingRecv error: recv got EOF from 127.0.0.1:47190 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:46:25.439356 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 6493
I20260619 16:46:25.444620 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 6627
I20260619 16:46:25.449744 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 6761
I20260619 16:46:25.455132 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 6399
2026-06-19T16:46:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2168 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:25 dist-test-slave-cgwx krb5kdc[6902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:25 dist-test-slave-cgwx krb5kdc[6902](info): set up 2 sockets
Jun 19 16:46:25 dist-test-slave-cgwx krb5kdc[6902](info): commencing operation
krb5kdc: starting...
W20260619 16:46:26.498975 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:26 dist-test-slave-cgwx krb5kdc[6902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:26.621794 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:42181
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:26.732726 6915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:26.733073 6915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:26.733148 6915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:26.737342 6915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:26.737437 6915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:26.737471 6915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:26.737507 6915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:26.737547 6915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:26.743088 6915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:42181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:26.744238 6915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:26.745366 6915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:26.754225 6920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:26.754215 6923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:26.754259 6921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:26.754544 6915 server_base.cc:1034] running on GCE node
I20260619 16:46:26.755139 6915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:26.756292 6915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:26.757452 6915 hybrid_clock.cc:648] HybridClock initialized: now 1781887586757436 us; error 34 us; skew 500 ppm
Jun 19 16:46:26 dist-test-slave-cgwx krb5kdc[6902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:26.765141 6915 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:26.766136 6915 webserver.cc:460] Webserver started at http://127.3.233.190:33321/ using document root <none> and password file <none>
I20260619 16:46:26.766773 6915 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:26.766842 6915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:26.767132 6915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:26.769243 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "7161f6c51b994d5f97044ea090232645"
format_stamp: "Formatted at 2026-06-19 16:46:26 on dist-test-slave-cgwx"
server_key: "1bbac0374118427daaf0493304eb9932"
server_key_iv: "26aab844b6179958d87835950adb65b4"
server_key_version: "encryptionkey@0"
I20260619 16:46:26.769836 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "7161f6c51b994d5f97044ea090232645"
format_stamp: "Formatted at 2026-06-19 16:46:26 on dist-test-slave-cgwx"
server_key: "1bbac0374118427daaf0493304eb9932"
server_key_iv: "26aab844b6179958d87835950adb65b4"
server_key_version: "encryptionkey@0"
I20260619 16:46:26.773874 6915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:46:26.776852 6930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:26.778214 6915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:26.778348 6915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "7161f6c51b994d5f97044ea090232645"
format_stamp: "Formatted at 2026-06-19 16:46:26 on dist-test-slave-cgwx"
server_key: "1bbac0374118427daaf0493304eb9932"
server_key_iv: "26aab844b6179958d87835950adb65b4"
server_key_version: "encryptionkey@0"
I20260619 16:46:26.778458 6915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:26.803750 6915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:26.807418 6915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:26.807638 6915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:26.814843 6915 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:42181
I20260619 16:46:26.815176 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:42181 every 8 connection(s)
I20260619 16:46:26.815912 6915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:26.818523 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 6915
I20260619 16:46:26.818677 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:26.818985 6983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:26.819010 4006 external_mini_cluster.cc:1428] Setting key 3190ea1d6b32685780da63192ec1b318
I20260619 16:46:26.825603 6983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645: Bootstrap starting.
Jun 19 16:46:26 dist-test-slave-cgwx krb5kdc[6902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:26.828248 6983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:26.829111 6983 log.cc:824] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:26.831480 6983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645: No bootstrap required, opened a new log
I20260619 16:46:26.833441 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.820319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.3.233.190:42181)
0619 16:46:26.820798 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:26.820814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:26.820856 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:46:26.821755 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:26.821780 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:26.822118 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:26.822474 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:26.823384 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:26.824388 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:26.825245 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:26.825761 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:26.825934 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:26.828673 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:26.828678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:26.828691 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:26.828719 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:26.830565 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:26.831076 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:26.831081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:26.831085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:26.831192 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:26.831477 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:26.831479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:26.831480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:26.831931 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:26.832073 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:26.832381 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:46:26.832693 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":141,"threads_started":1}
I20260619 16:46:26.835475 6983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7161f6c51b994d5f97044ea090232645" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 42181 } }
I20260619 16:46:26.835769 6983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:26.835843 6983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7161f6c51b994d5f97044ea090232645, State: Initialized, Role: FOLLOWER
I20260619 16:46:26.836355 6983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7161f6c51b994d5f97044ea090232645" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 42181 } }
I20260619 16:46:26.836516 6983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:26.836602 6983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:26.836705 6983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:26.836730 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:26.838053 6983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7161f6c51b994d5f97044ea090232645" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 42181 } }
I20260619 16:46:26.838457 6983 leader_election.cc:302] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7161f6c51b994d5f97044ea090232645; no voters:
I20260619 16:46:26.838840 6983 leader_election.cc:288] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:26.838950 6989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:26.839126 6989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [term 1 LEADER]: Becoming Leader. State: Replica: 7161f6c51b994d5f97044ea090232645, State: Running, Role: LEADER
I20260619 16:46:26.839504 6989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7161f6c51b994d5f97044ea090232645" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 42181 } }
I20260619 16:46:26.840195 6983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:26.841074 6991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7161f6c51b994d5f97044ea090232645. Latest consensus state: current_term: 1 leader_uuid: "7161f6c51b994d5f97044ea090232645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7161f6c51b994d5f97044ea090232645" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 42181 } } }
I20260619 16:46:26.841300 6991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:26.841255 6990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7161f6c51b994d5f97044ea090232645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7161f6c51b994d5f97044ea090232645" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 42181 } } }
I20260619 16:46:26.841406 6990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:26.841818 6998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:26.845220 6998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:26.852020 6998 catalog_manager.cc:1347] Generated new cluster ID: e447392ddd6e44248c47249a5245f659
I20260619 16:46:26.852094 6998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:26.855863 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:26.851025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.3.233.190:42181)
0619 16:46:26.851138 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:26.851142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:26.851432 (+ 290us) server_negotiation.cc:407] Connection header received
0619 16:46:26.851565 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:26.851568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:26.851635 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:26.851729 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:26.852318 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:26.852857 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:26.853718 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:26.853979 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:26.854088 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:26.854963 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:26.854967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:26.854970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:26.855401 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:26.855428 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:26.855443 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:26.855540 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:26.855628 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 16:46:26.855698 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:46:26.864482 6998 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:26.865713 6998 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:26.880844 6998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7161f6c51b994d5f97044ea090232645: Generated new TSK 0
I20260619 16:46:26.881731 6998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:26.889659 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 6915
2026-06-19T16:46:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1432 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:26 dist-test-slave-cgwx krb5kdc[7016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:26 dist-test-slave-cgwx krb5kdc[7016](info): set up 2 sockets
Jun 19 16:46:26 dist-test-slave-cgwx krb5kdc[7016](info): commencing operation
krb5kdc: starting...
W20260619 16:46:27.938913 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:28.066079 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38497
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:28.178521 7029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:28.178807 7029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:28.178849 7029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:28.182907 7029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:28.182987 7029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:28.183007 7029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:28.183055 7029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:28.183090 7029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:28.188364 7029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:28.189543 7029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:28.190557 7029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:28.199000 7035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:28.199165 7029 server_base.cc:1034] running on GCE node
W20260619 16:46:28.199070 7034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.199005 7037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:28.199857 7029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:28.200947 7029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:28.202102 7029 hybrid_clock.cc:648] HybridClock initialized: now 1781887588202082 us; error 35 us; skew 500 ppm
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:28.204950 7029 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:28.205870 7029 webserver.cc:460] Webserver started at http://127.3.233.190:43041/ using document root <none> and password file <none>
I20260619 16:46:28.206388 7029 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:28.206445 7029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:28.206619 7029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:28.208731 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "00add07dd5eb4895b72bb02475f32e78"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "e46114112f35155f2ea06846e0422982"
server_key_iv: "9dc893247f70bef4600795e865c3a712"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.209452 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "00add07dd5eb4895b72bb02475f32e78"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "e46114112f35155f2ea06846e0422982"
server_key_iv: "9dc893247f70bef4600795e865c3a712"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.213357 7029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 16:46:28.215837 7044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.217046 7029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:46:28.217173 7029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "00add07dd5eb4895b72bb02475f32e78"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "e46114112f35155f2ea06846e0422982"
server_key_iv: "9dc893247f70bef4600795e865c3a712"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.217278 7029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:28.251078 7029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:28.254021 7029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:28.254230 7029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:28.261035 7029 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:38497
I20260619 16:46:28.261344 7096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:38497 every 8 connection(s)
I20260619 16:46:28.262007 7029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:28.262660 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7029
I20260619 16:46:28.262781 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:28.263110 4006 external_mini_cluster.cc:1428] Setting key ce4b3e3b051f3f75048a426cca6803a8
I20260619 16:46:28.265280 7097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:28.271771 7097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78: Bootstrap starting.
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:28.273878 7097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:28.274561 7097 log.cc:824] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:28.276464 7097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78: No bootstrap required, opened a new log
I20260619 16:46:28.277060 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.264417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.3.233.190:38497)
0619 16:46:28.264799 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:28.264825 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:28.264859 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:46:28.265542 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:28.265561 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:28.265905 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:28.266180 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:28.267112 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.268090 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.269237 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.269680 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.269788 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.272534 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:28.272539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:28.272552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:28.272577 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:28.274320 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.274771 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.274776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.274780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.274882 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.275203 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.275206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.275216 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.275573 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:28.275767 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:28.276066 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 16:46:28.276336 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20260619 16:46:28.280318 7097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00add07dd5eb4895b72bb02475f32e78" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38497 } }
I20260619 16:46:28.280548 7097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:28.280627 7097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00add07dd5eb4895b72bb02475f32e78, State: Initialized, Role: FOLLOWER
I20260619 16:46:28.281081 7097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00add07dd5eb4895b72bb02475f32e78" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38497 } }
I20260619 16:46:28.281230 7097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:28.281317 7097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:28.281410 7097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:28.282264 7097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00add07dd5eb4895b72bb02475f32e78" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38497 } }
I20260619 16:46:28.282619 7097 leader_election.cc:302] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00add07dd5eb4895b72bb02475f32e78; no voters:
I20260619 16:46:28.282984 7097 leader_election.cc:288] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:28.283151 7102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:28.283434 7102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [term 1 LEADER]: Becoming Leader. State: Replica: 00add07dd5eb4895b72bb02475f32e78, State: Running, Role: LEADER
I20260619 16:46:28.283741 7102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00add07dd5eb4895b72bb02475f32e78" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38497 } }
I20260619 16:46:28.284217 7097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:28.285542 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00add07dd5eb4895b72bb02475f32e78. Latest consensus state: current_term: 1 leader_uuid: "00add07dd5eb4895b72bb02475f32e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00add07dd5eb4895b72bb02475f32e78" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38497 } } }
I20260619 16:46:28.285579 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00add07dd5eb4895b72bb02475f32e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00add07dd5eb4895b72bb02475f32e78" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38497 } } }
I20260619 16:46:28.285666 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:28.285691 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:28.286053 7112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:28.289445 7112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:28.294749 7112 catalog_manager.cc:1347] Generated new cluster ID: b83ae2dd5a3541feae9d4db0ca252f8c
I20260619 16:46:28.294816 7112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:28.306345 7112 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:28.307346 7112 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:28.314646 7112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00add07dd5eb4895b72bb02475f32e78: Generated new TSK 0
I20260619 16:46:28.315472 7112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:28.353926 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38497
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:28.492213 7123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:28.492480 7123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:28.492523 7123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:28.496568 7123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:28.496629 7123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:28.496694 7123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:28.502058 7123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:28.503082 7123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:28.504055 7123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:28.513653 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.513744 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:28.513861 7123 server_base.cc:1034] running on GCE node
W20260619 16:46:28.514053 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:28.514636 7123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:28.515408 7123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:28.516582 7123 hybrid_clock.cc:648] HybridClock initialized: now 1781887588516563 us; error 33 us; skew 500 ppm
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:28.519431 7123 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:28.520331 7123 webserver.cc:460] Webserver started at http://127.3.233.129:45753/ using document root <none> and password file <none>
I20260619 16:46:28.520874 7123 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:28.520956 7123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:28.521173 7123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:28.523262 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "b93465bea92b401ab58440ffd062b41e"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "d5c41de142e0eb5a9911c70390e13700"
server_key_iv: "821a0dc40c65f9d6392133c07d90f7cb"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.523785 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "b93465bea92b401ab58440ffd062b41e"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "d5c41de142e0eb5a9911c70390e13700"
server_key_iv: "821a0dc40c65f9d6392133c07d90f7cb"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.527405 7123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:28.529749 7138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.531123 7123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:28.531216 7123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "b93465bea92b401ab58440ffd062b41e"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "d5c41de142e0eb5a9911c70390e13700"
server_key_iv: "821a0dc40c65f9d6392133c07d90f7cb"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.531309 7123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:28.551934 7123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:28.554883 7123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:28.555058 7123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:28.555577 7123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:28.556334 7123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:28.556377 7123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.556416 7123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:28.556432 7123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.565160 7123 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:40837
I20260619 16:46:28.565182 7251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:40837 every 8 connection(s)
I20260619 16:46:28.566082 7123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:28.571154 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7123
I20260619 16:46:28.571277 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:28.571583 4006 external_mini_cluster.cc:1428] Setting key ffee37cb68cac170b33bed29bacb1d2a
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:28.580804 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.567957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:51975 (local address 127.3.233.190:38497)
0619 16:46:28.568122 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:28.568127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:28.568928 (+ 801us) server_negotiation.cc:407] Connection header received
0619 16:46:28.570315 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:28.570318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:28.570374 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:28.570461 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:28.571914 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.572558 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.573866 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.574106 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.574222 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.577141 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:28.577145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:28.577148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:28.577172 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:28.578631 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.579248 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.579252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.579254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.579325 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.579673 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.579680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.579681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.579849 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:28.579956 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:28.580501 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 16:46:28.580620 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:46:28.581640 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.568319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38497 (local address 127.3.233.129:51975)
0619 16:46:28.568758 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.568807 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:28.570097 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:28.570613 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:28.570620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:28.571046 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:28.571752 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.571773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.572692 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.572696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.573714 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.573724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.574257 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.574261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.574436 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.575100 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:28.575127 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:28.576957 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:28.578768 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.578774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.578788 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.579125 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.579416 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.579420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.579422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.579563 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.579953 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:28.579960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:28.580257 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 16:46:28.580448 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 16:46:28.580770 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:28.583086 7252 heartbeater.cc:347] Connected to a master server at 127.3.233.190:38497
I20260619 16:46:28.583334 7252 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:28.583884 7252 heartbeater.cc:511] Master 127.3.233.190:38497 requested a full tablet report, sending...
I20260619 16:46:28.585687 7061 ts_manager.cc:194] Registered new tserver with Master: b93465bea92b401ab58440ffd062b41e (127.3.233.129:40837)
I20260619 16:46:28.586895 7061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:51975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:28.597818 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38497
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:28.708797 7257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:28.709113 7257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:28.709189 7257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:28.713429 7257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:28.713526 7257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:28.713624 7257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:28.718906 7257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:28.720091 7257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:28.721184 7257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:28.731001 7265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.731012 7263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.731038 7262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:28.731282 7257 server_base.cc:1034] running on GCE node
I20260619 16:46:28.731848 7257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:28.732564 7257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:28.733739 7257 hybrid_clock.cc:648] HybridClock initialized: now 1781887588733722 us; error 38 us; skew 500 ppm
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:28.736635 7257 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:28.737555 7257 webserver.cc:460] Webserver started at http://127.3.233.130:34801/ using document root <none> and password file <none>
I20260619 16:46:28.738111 7257 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:28.738173 7257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:28.738410 7257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:28.740561 7257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "f4e537e6defd42ba85c991222d99159b"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "cb0aa1ac751398cfdae813054d770fc2"
server_key_iv: "9f47d8afab06b1e03832ce1ea08913ee"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.741122 7257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "f4e537e6defd42ba85c991222d99159b"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "cb0aa1ac751398cfdae813054d770fc2"
server_key_iv: "9f47d8afab06b1e03832ce1ea08913ee"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.744975 7257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:28.747581 7272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.748910 7257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:28.749047 7257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "f4e537e6defd42ba85c991222d99159b"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "cb0aa1ac751398cfdae813054d770fc2"
server_key_iv: "9f47d8afab06b1e03832ce1ea08913ee"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.749164 7257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:28.765491 7257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:28.768559 7257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:28.768764 7257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:28.769372 7257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:28.770170 7257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:28.770246 7257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.770314 7257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:28.770362 7257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.779567 7257 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:35187
I20260619 16:46:28.779584 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:35187 every 8 connection(s)
I20260619 16:46:28.780450 7257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:28.784121 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7257
I20260619 16:46:28.784221 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:28.784509 4006 external_mini_cluster.cc:1428] Setting key e1208b865f39b2e5f0c2392f675d25e8
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:46:28.797866 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.782274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:56859 (local address 127.3.233.190:38497)
0619 16:46:28.782424 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:28.782429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:28.783429 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 16:46:28.785317 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:28.785321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:28.785379 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:28.785468 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:28.787163 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.788018 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.789947 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.790380 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.790522 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.793647 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:28.793654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:28.793658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:28.793689 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:28.795512 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.796064 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.796071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.796072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.796149 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:28.796585 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:28.796588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:28.796589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:28.796787 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:28.796867 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:28.797517 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 16:46:28.797642 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:46:28.798702 7388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:28.782690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38497 (local address 127.3.233.130:56859)
0619 16:46:28.783257 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:28.783304 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:28.785048 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:28.785690 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:28.785700 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:28.786280 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:28.786999 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.787025 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.788180 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.788185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.789801 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:28.789812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:28.790534 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:28.790537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:28.790808 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:28.791533 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:28.791550 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:28.793435 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:28.795655 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.795660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.795676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.795962 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.796312 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:28.796318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:28.796323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:28.796490 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:28.796911 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:28.796925 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:28.797254 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 16:46:28.797439 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:46:28.797792 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":118,"threads_started":1}
I20260619 16:46:28.799995 7386 heartbeater.cc:347] Connected to a master server at 127.3.233.190:38497
I20260619 16:46:28.800314 7386 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:28.800915 7386 heartbeater.cc:511] Master 127.3.233.190:38497 requested a full tablet report, sending...
I20260619 16:46:28.802037 7061 ts_manager.cc:194] Registered new tserver with Master: f4e537e6defd42ba85c991222d99159b (127.3.233.130:35187)
I20260619 16:46:28.802641 7061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:56859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:28.814280 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38497
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:28.930163 7391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:28.930441 7391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:28.930490 7391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:28.934900 7391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:28.934981 7391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:28.935041 7391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:28.940500 7391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:28.941519 7391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:28.942607 7391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:28.952749 7397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.952878 7396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:28.952803 7399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:28.953568 7391 server_base.cc:1034] running on GCE node
I20260619 16:46:28.954119 7391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:28.955081 7391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:28.956254 7391 hybrid_clock.cc:648] HybridClock initialized: now 1781887588956175 us; error 115 us; skew 500 ppm
Jun 19 16:46:28 dist-test-slave-cgwx krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:28.959807 7391 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:28.960764 7391 webserver.cc:460] Webserver started at http://127.3.233.131:43025/ using document root <none> and password file <none>
I20260619 16:46:28.961349 7391 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:28.961409 7391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:28.961637 7391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:28.963805 7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "34b08fec60d24c20acce4fa49cc11917"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "8f288c0f4284ef490e99d562f497e8ed"
server_key_iv: "e07817b5764a7b25ddc770d5a1ea3b3e"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.964388 7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "34b08fec60d24c20acce4fa49cc11917"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "8f288c0f4284ef490e99d562f497e8ed"
server_key_iv: "e07817b5764a7b25ddc770d5a1ea3b3e"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.968248 7391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:46:28.971264 7406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.972512 7391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 16:46:28.972640 7391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "34b08fec60d24c20acce4fa49cc11917"
format_stamp: "Formatted at 2026-06-19 16:46:28 on dist-test-slave-cgwx"
server_key: "8f288c0f4284ef490e99d562f497e8ed"
server_key_iv: "e07817b5764a7b25ddc770d5a1ea3b3e"
server_key_version: "encryptionkey@0"
I20260619 16:46:28.972759 7391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:28.990383 7391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:28.993844 7391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:28.994052 7391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:28.994656 7391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:28.995597 7391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:28.995678 7391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:28.995745 7391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:28.995779 7391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:29.005137 7391 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:37747
I20260619 16:46:29.005281 7519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:37747 every 8 connection(s)
I20260619 16:46:29.006232 7391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:29.011132 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7391
I20260619 16:46:29.011281 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:29.011579 4006 external_mini_cluster.cc:1428] Setting key a502a62568aec56324b3ff48debdc2c7
Jun 19 16:46:29 dist-test-slave-cgwx krb5kdc[7016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:29.020964 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:29.008102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:58259 (local address 127.3.233.190:38497)
0619 16:46:29.008236 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:29.008243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:29.009153 (+ 910us) server_negotiation.cc:407] Connection header received
0619 16:46:29.010549 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:29.010552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:29.010597 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:29.010675 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:29.012097 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:29.012869 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:29.014532 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:29.014744 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:29.014867 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:29.017492 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:29.017495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:29.017497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:29.017521 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:29.019115 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:29.019572 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:29.019575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:29.019576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:29.019650 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:29.019945 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:29.019948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:29.019949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:29.020121 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:29.020195 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:29.020691 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 16:46:29.020821 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:46:29.021899 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:29.008516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38497 (local address 127.3.233.131:58259)
0619 16:46:29.008998 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:29.009042 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:29.010343 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:29.010862 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:29.010869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:29.011329 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:29.011928 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:29.011950 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:29.013064 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:29.013069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:29.014405 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:29.014421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:29.014986 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:29.014991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:29.015174 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:29.015838 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:29.015854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:29.017335 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:29.019213 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:29.019218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:29.019233 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:29.019486 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:29.019734 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:29.019736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:29.019737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:29.019864 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:29.020205 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:29.020210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:29.020484 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 16:46:29.020643 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 16:46:29.020968 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260619 16:46:29.023175 7520 heartbeater.cc:347] Connected to a master server at 127.3.233.190:38497
I20260619 16:46:29.023427 7520 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:29.023914 7520 heartbeater.cc:511] Master 127.3.233.190:38497 requested a full tablet report, sending...
I20260619 16:46:29.025059 7061 ts_manager.cc:194] Registered new tserver with Master: 34b08fec60d24c20acce4fa49cc11917 (127.3.233.131:37747)
I20260619 16:46:29.025666 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:29.025816 7061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:58259
Jun 19 16:46:29 dist-test-slave-cgwx krb5kdc[7016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887589, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 16:46:29 dist-test-slave-cgwx krb5kdc[7016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887589, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:29.052554 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:29.043659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.3.233.190:38497)
0619 16:46:29.043853 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:29.043858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:29.043992 (+ 134us) server_negotiation.cc:407] Connection header received
0619 16:46:29.044177 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:29.044180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:29.044224 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:29.044309 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:29.045053 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:29.045579 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:29.046434 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:29.046659 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:29.046764 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:29.049289 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:29.049293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:29.049300 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:29.049327 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:29.050701 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:29.051273 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:29.051277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:29.051279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:29.051365 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:29.051643 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:29.051646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:29.051648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:29.051883 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:29.052028 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:29.052266 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 16:46:29.052382 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 16:46:29.052986 7061 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37374
I20260619 16:46:29.054358 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7123
I20260619 16:46:29.059816 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7257
I20260619 16:46:29.065254 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7391
I20260619 16:46:29.070444 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7029
2026-06-19T16:46:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2183 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:29 dist-test-slave-cgwx krb5kdc[7532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:29 dist-test-slave-cgwx krb5kdc[7532](info): set up 2 sockets
Jun 19 16:46:29 dist-test-slave-cgwx krb5kdc[7532](info): commencing operation
krb5kdc: starting...
W20260619 16:46:30.123028 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:30 dist-test-slave-cgwx krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:30.244199 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:36613
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:30.355803 7545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:30.356115 7545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:30.356199 7545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:30.360198 7545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:30.360294 7545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:30.360329 7545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:30.360368 7545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:30.360400 7545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:30.365643 7545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:30.366724 7545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:30.367758 7545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:30.375998 7550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:30.376016 7551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:30.376152 7545 server_base.cc:1034] running on GCE node
W20260619 16:46:30.376291 7553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:30.376771 7545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:30.377871 7545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:30.379047 7545 hybrid_clock.cc:648] HybridClock initialized: now 1781887590379029 us; error 31 us; skew 500 ppm
Jun 19 16:46:30 dist-test-slave-cgwx krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887590, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:30.381743 7545 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:30.382664 7545 webserver.cc:460] Webserver started at http://127.3.233.190:40961/ using document root <none> and password file <none>
I20260619 16:46:30.383271 7545 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:30.383358 7545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:30.383579 7545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:30.385551 7545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "9e9860af09844c5bbdbd80b378ed4e0f"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "d9795bd2e6cab9dd1a5ce3eadc8c89e2"
server_key_iv: "d7d77b077f6673189b51495bec189f7e"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.386106 7545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "9e9860af09844c5bbdbd80b378ed4e0f"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "d9795bd2e6cab9dd1a5ce3eadc8c89e2"
server_key_iv: "d7d77b077f6673189b51495bec189f7e"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.389729 7545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 16:46:30.392074 7560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:30.393363 7545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:30.393496 7545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "9e9860af09844c5bbdbd80b378ed4e0f"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "d9795bd2e6cab9dd1a5ce3eadc8c89e2"
server_key_iv: "d7d77b077f6673189b51495bec189f7e"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.393602 7545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:30.412541 7545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:30.415987 7545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:30.416195 7545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:30.422957 7545 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:36613
I20260619 16:46:30.423022 7612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:36613 every 8 connection(s)
I20260619 16:46:30.423950 7545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:30.426643 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:30.431946 7613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f: Bootstrap starting.
I20260619 16:46:30.433297 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7545
I20260619 16:46:30.433462 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:30.433790 4006 external_mini_cluster.cc:1428] Setting key f35371f8cce093f73076c9c0f6a6a3c8
I20260619 16:46:30.434618 7613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:30.435568 7613 log.cc:824] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:30.437968 7613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f: No bootstrap required, opened a new log
I20260619 16:46:30.442322 7613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 36613 } }
I20260619 16:46:30.442548 7613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:30.442582 7613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9860af09844c5bbdbd80b378ed4e0f, State: Initialized, Role: FOLLOWER
I20260619 16:46:30.443137 7613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 36613 } }
Jun 19 16:46:30 dist-test-slave-cgwx krb5kdc[7532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:30.443303 7613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:30.443395 7613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:30.443521 7613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:30.444823 7613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 36613 } }
I20260619 16:46:30.445209 7613 leader_election.cc:302] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e9860af09844c5bbdbd80b378ed4e0f; no voters:
I20260619 16:46:30.445583 7613 leader_election.cc:288] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:30.445703 7618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:30.445950 7618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [term 1 LEADER]: Becoming Leader. State: Replica: 9e9860af09844c5bbdbd80b378ed4e0f, State: Running, Role: LEADER
I20260619 16:46:30.446283 7618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 36613 } }
I20260619 16:46:30.447042 7613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:30.448031 7620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9860af09844c5bbdbd80b378ed4e0f. Latest consensus state: current_term: 1 leader_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 36613 } } }
I20260619 16:46:30.448160 7620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:30.448555 7619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9860af09844c5bbdbd80b378ed4e0f" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 36613 } } }
I20260619 16:46:30.448663 7619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:30.448772 7622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:30.450172 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.434967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.3.233.190:36613)
0619 16:46:30.435510 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:30.435529 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:30.435572 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:46:30.436347 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:30.436366 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:30.436785 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:30.437154 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:30.437849 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.439302 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.440187 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.440981 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.441180 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:30.443937 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:30.443943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:30.443960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:30.443992 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:30.446194 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:30.446741 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:30.446746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:30.446753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:30.446907 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:30.447267 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:30.447270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:30.447272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:30.447898 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:30.448119 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:30.448799 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 16:46:30.449112 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":381,"thread_start_us":164,"threads_started":1}
I20260619 16:46:30.453538 7622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:30.459443 7622 catalog_manager.cc:1347] Generated new cluster ID: 2b897a91594f434e9f333281df123c8b
I20260619 16:46:30.459530 7622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:30.472486 7622 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:30.473517 7622 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:30.479566 7622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e9860af09844c5bbdbd80b378ed4e0f: Generated new TSK 0
I20260619 16:46:30.480309 7622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:30.514120 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:36613
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:30.635537 7639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:30.635833 7639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:30.635924 7639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:30.640094 7639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:30.640183 7639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:30.640275 7639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:30.645661 7639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:30.646847 7639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:30.647992 7639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:30.657886 7645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:30.657905 7644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:30.657876 7647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:30.658223 7639 server_base.cc:1034] running on GCE node
I20260619 16:46:30.658684 7639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:30.659405 7639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:30.660591 7639 hybrid_clock.cc:648] HybridClock initialized: now 1781887590660559 us; error 52 us; skew 500 ppm
Jun 19 16:46:30 dist-test-slave-cgwx krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887590, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:30.663425 7639 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:30.664325 7639 webserver.cc:460] Webserver started at http://127.3.233.129:43537/ using document root <none> and password file <none>
I20260619 16:46:30.664901 7639 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:30.664994 7639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:30.665220 7639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:30.667325 7639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "f962227673054fa894a51ad509fe22fb"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "86d63c0924bfbd14c53403b5b081c4e6"
server_key_iv: "d670d8448bff7ead29f3ca57cfbac66c"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.667894 7639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "f962227673054fa894a51ad509fe22fb"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "86d63c0924bfbd14c53403b5b081c4e6"
server_key_iv: "d670d8448bff7ead29f3ca57cfbac66c"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.671717 7639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:30.674356 7654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:30.675712 7639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:30.675848 7639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "f962227673054fa894a51ad509fe22fb"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "86d63c0924bfbd14c53403b5b081c4e6"
server_key_iv: "d670d8448bff7ead29f3ca57cfbac66c"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.675963 7639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:30.699215 7639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:30.702456 7639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:30.702610 7639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:30.703248 7639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:30.704022 7639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:30.704068 7639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:30.704107 7639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:30.704123 7639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:30.713065 7639 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:44595
I20260619 16:46:30.713090 7767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:44595 every 8 connection(s)
I20260619 16:46:30.713997 7639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:30.721392 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7639
I20260619 16:46:30.721551 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:30.721908 4006 external_mini_cluster.cc:1428] Setting key acfc16230e95973eef1e299f9aabeecc
Jun 19 16:46:30 dist-test-slave-cgwx krb5kdc[7532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887590, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:30.728965 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.716007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:60359 (local address 127.3.233.190:36613)
0619 16:46:30.716152 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:30.716155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:30.716991 (+ 836us) server_negotiation.cc:407] Connection header received
0619 16:46:30.718387 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:30.718391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:30.718450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:30.718551 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:30.719820 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.720348 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.721507 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.721822 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.721934 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:30.725293 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:30.725296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:30.725298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:30.725322 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:30.726810 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:30.727397 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:30.727402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:30.727404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:30.727488 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:30.727851 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:30.727855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:30.727857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:30.728047 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:30.728149 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:30.728675 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 16:46:30.728803 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:30.729903 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.716329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:36613 (local address 127.3.233.129:60359)
0619 16:46:30.716834 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:30.716876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:30.718162 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:30.718740 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:30.718747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:30.719174 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:30.719656 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:30.719669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.720472 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.720481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:30.721379 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:30.721388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.721972 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.721976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:30.722149 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:30.723068 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:30.723094 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:30.725072 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:30.726975 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:30.726981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:30.727005 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:30.727289 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:30.727592 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:30.727595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:30.727597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:30.727740 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:30.728160 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:30.728168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:30.728433 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 16:46:30.728622 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 16:46:30.728978 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260619 16:46:30.731299 7768 heartbeater.cc:347] Connected to a master server at 127.3.233.190:36613
I20260619 16:46:30.731580 7768 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:30.732206 7768 heartbeater.cc:511] Master 127.3.233.190:36613 requested a full tablet report, sending...
I20260619 16:46:30.733847 7577 ts_manager.cc:194] Registered new tserver with Master: f962227673054fa894a51ad509fe22fb (127.3.233.129:44595)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:30.735172 7577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:60359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:30.750900 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:36613
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:30.863040 7773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:30.863320 7773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:30.863389 7773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:30.867606 7773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:30.867669 7773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:30.867758 7773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:30.873577 7773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:30.874744 7773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:30.875815 7773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:30.885756 7781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:30.885764 7778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:30.885747 7779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:30.885963 7773 server_base.cc:1034] running on GCE node
I20260619 16:46:30.886523 7773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:30.887240 7773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:30.888420 7773 hybrid_clock.cc:648] HybridClock initialized: now 1781887590888385 us; error 47 us; skew 500 ppm
Jun 19 16:46:30 dist-test-slave-cgwx krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887590, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:30.891250 7773 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:30.892220 7773 webserver.cc:460] Webserver started at http://127.3.233.130:32889/ using document root <none> and password file <none>
I20260619 16:46:30.892746 7773 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:30.892807 7773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:30.892969 7773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:30.895057 7773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "3fc5af2877a74b6bbb814e2a67429e0e"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "6225d589a8bbef1aea0b529a9a889ea4"
server_key_iv: "8de4584f95e7d40c517aa8a39d7da2e2"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.895639 7773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "3fc5af2877a74b6bbb814e2a67429e0e"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "6225d589a8bbef1aea0b529a9a889ea4"
server_key_iv: "8de4584f95e7d40c517aa8a39d7da2e2"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.899667 7773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:30.902235 7788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:30.903891 7773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:30.904002 7773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "3fc5af2877a74b6bbb814e2a67429e0e"
format_stamp: "Formatted at 2026-06-19 16:46:30 on dist-test-slave-cgwx"
server_key: "6225d589a8bbef1aea0b529a9a889ea4"
server_key_iv: "8de4584f95e7d40c517aa8a39d7da2e2"
server_key_version: "encryptionkey@0"
I20260619 16:46:30.904098 7773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:30.919423 7773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:30.922704 7773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:30.922888 7773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:30.923521 7773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:30.924396 7773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:30.924446 7773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:30.924491 7773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:30.924508 7773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:30.934476 7773 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:41563
I20260619 16:46:30.934496 7901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:41563 every 8 connection(s)
I20260619 16:46:30.935510 7773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:30.937377 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7773
I20260619 16:46:30.937512 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:30.937857 4006 external_mini_cluster.cc:1428] Setting key 480fffa38291c530c02178b0b0a2b48e
Jun 19 16:46:30 dist-test-slave-cgwx krb5kdc[7532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887590, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:46:30.952204 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.937498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:43167 (local address 127.3.233.190:36613)
0619 16:46:30.937619 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:30.937623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:30.938807 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 16:46:30.940416 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:30.940420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:30.940469 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:30.940541 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:30.942138 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.942707 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.944644 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.944910 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.945005 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:30.948034 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:30.948046 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:30.948048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:30.948070 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:30.949597 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:30.950176 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:30.950180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:30.950182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:30.950291 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:30.950670 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:30.950676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:30.950678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:30.950908 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:30.951052 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:30.951653 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 16:46:30.951850 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:46:30.953136 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:30.938048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:36613 (local address 127.3.233.130:43167)
0619 16:46:30.938597 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:30.938641 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:30.940202 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:30.940715 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:30.940722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:30.941231 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:30.941950 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:30.941977 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.942846 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.942853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:30.944504 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:30.944518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:30.945089 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:30.945095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:30.945352 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:30.946006 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:30.946028 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:30.947864 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:30.949720 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:30.949726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:30.949742 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:30.950084 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:30.950383 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:30.950386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:30.950388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:30.950564 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:30.951056 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:30.951061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:30.951302 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 16:46:30.951528 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 16:46:30.951926 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":163,"threads_started":1}
I20260619 16:46:30.956118 7902 heartbeater.cc:347] Connected to a master server at 127.3.233.190:36613
I20260619 16:46:30.956411 7902 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:30.957051 7902 heartbeater.cc:511] Master 127.3.233.190:36613 requested a full tablet report, sending...
I20260619 16:46:30.958498 7577 ts_manager.cc:194] Registered new tserver with Master: 3fc5af2877a74b6bbb814e2a67429e0e (127.3.233.130:41563)
I20260619 16:46:30.959321 7577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:43167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:30.970485 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:36613
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:31.082209 7907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:31.082487 7907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:31.082561 7907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:31.086622 7907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:31.086720 7907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:31.086810 7907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:31.092334 7907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:31.093468 7907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:31.094515 7907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:31.104203 7913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:31.104282 7915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:31.104388 7907 server_base.cc:1034] running on GCE node
W20260619 16:46:31.104210 7912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:31.104827 7907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:31.105463 7907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:31.106639 7907 hybrid_clock.cc:648] HybridClock initialized: now 1781887591106614 us; error 36 us; skew 500 ppm
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887591, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:31.109431 7907 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:31.110260 7907 webserver.cc:460] Webserver started at http://127.3.233.131:44163/ using document root <none> and password file <none>
I20260619 16:46:31.110764 7907 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:31.110821 7907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:31.111047 7907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:31.112982 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "5265c1d5ae1d4883a5762e38b59723a4"
format_stamp: "Formatted at 2026-06-19 16:46:31 on dist-test-slave-cgwx"
server_key: "1385db829d30aff54da20c7977309e9b"
server_key_iv: "6de2bbb91840cdba4b7f16b721f6c3c5"
server_key_version: "encryptionkey@0"
I20260619 16:46:31.113482 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "5265c1d5ae1d4883a5762e38b59723a4"
format_stamp: "Formatted at 2026-06-19 16:46:31 on dist-test-slave-cgwx"
server_key: "1385db829d30aff54da20c7977309e9b"
server_key_iv: "6de2bbb91840cdba4b7f16b721f6c3c5"
server_key_version: "encryptionkey@0"
I20260619 16:46:31.117095 7907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:46:31.119519 7922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:31.120714 7907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:31.120805 7907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "5265c1d5ae1d4883a5762e38b59723a4"
format_stamp: "Formatted at 2026-06-19 16:46:31 on dist-test-slave-cgwx"
server_key: "1385db829d30aff54da20c7977309e9b"
server_key_iv: "6de2bbb91840cdba4b7f16b721f6c3c5"
server_key_version: "encryptionkey@0"
I20260619 16:46:31.120887 7907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:31.148041 7907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:31.151789 7907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:31.151955 7907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:31.152498 7907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:31.153353 7907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:31.153400 7907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:31.153440 7907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:31.153455 7907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:31.162703 7907 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:40865
I20260619 16:46:31.162741 8035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:40865 every 8 connection(s)
I20260619 16:46:31.163645 7907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:31.167236 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 7907
I20260619 16:46:31.167379 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:31.167697 4006 external_mini_cluster.cc:1428] Setting key 39aff1a8b71a85df678826535d1ab4b1
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[7532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887591, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:31.177534 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:31.165454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:34461 (local address 127.3.233.190:36613)
0619 16:46:31.165619 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:31.165623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:31.166439 (+ 816us) server_negotiation.cc:407] Connection header received
0619 16:46:31.168222 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:31.168226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:31.168285 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:31.168368 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:31.169607 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.170135 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.171354 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.171574 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.171694 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:31.174164 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:31.174167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:31.174169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:31.174200 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:31.175691 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:31.176166 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:31.176168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:31.176169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:31.176240 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:31.176541 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:31.176544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:31.176545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:31.176720 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:31.176798 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:31.177314 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 16:46:31.177413 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:46:31.178368 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:31.165852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:36613 (local address 127.3.233.131:34461)
0619 16:46:31.166286 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:31.166327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:31.167967 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:31.168584 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:31.168593 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:31.168989 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:31.169467 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:31.169480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.170260 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.170264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:31.171235 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:31.171244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.171723 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.171727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:31.171899 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:31.172442 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:31.172458 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:31.173992 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:31.175796 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:31.175800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:31.175812 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:31.176081 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:31.176335 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:31.176337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:31.176338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:31.176464 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:31.176833 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:31.176838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:31.177080 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 16:46:31.177255 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 16:46:31.177506 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260619 16:46:31.179540 8036 heartbeater.cc:347] Connected to a master server at 127.3.233.190:36613
I20260619 16:46:31.179781 8036 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:31.180339 8036 heartbeater.cc:511] Master 127.3.233.190:36613 requested a full tablet report, sending...
I20260619 16:46:31.181391 7576 ts_manager.cc:194] Registered new tserver with Master: 5265c1d5ae1d4883a5762e38b59723a4 (127.3.233.131:40865)
I20260619 16:46:31.181963 7576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:34461
I20260619 16:46:31.193373 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[7532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:31.220988 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:31.211178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.3.233.129:44595)
0619 16:46:31.211496 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:31.211511 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:31.211538 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:46:31.211597 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:31.211602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:31.211834 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:31.211964 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:31.212753 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.213575 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.214483 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.214900 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.215104 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:31.217716 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:31.217725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:31.217740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:31.217765 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:31.219404 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:31.219800 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:31.219805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:31.219806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:31.219883 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:31.220110 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:31.220112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:31.220114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:31.220354 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:31.220529 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:31.220711 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 16:46:31.220832 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":111,"threads_started":1}
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[7532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:31.232138 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:31.224094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.3.233.190:36613)
0619 16:46:31.224206 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:31.224209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:31.224244 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:46:31.224405 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:31.224407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:31.224449 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:31.224517 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:31.225073 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.225662 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.226406 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:31.226613 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:31.226689 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:31.228998 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:31.229001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:31.229003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:31.229024 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:31.230455 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:31.230997 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:31.231000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:31.231001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:31.231070 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:31.231324 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:31.231326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:31.231327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:31.231518 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:31.231627 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:31.231862 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:46:31.231949 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 16:46:31.232575 7576 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40456
I20260619 16:46:31.233289 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7639
I20260619 16:46:31.238840 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7773
I20260619 16:46:31.244295 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7907
I20260619 16:46:31.249951 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 7545
2026-06-19T16:46:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2177 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[8048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[8048](info): set up 2 sockets
Jun 19 16:46:31 dist-test-slave-cgwx krb5kdc[8048](info): commencing operation
krb5kdc: starting...
W20260619 16:46:32.303195 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:32 dist-test-slave-cgwx krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:32.423497 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34943
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:32.532070 8061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:32.532378 8061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:32.532457 8061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:32.536438 8061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:32.536526 8061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:32.536563 8061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:32.536602 8061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:32.536636 8061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:32.541849 8061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:32.542960 8061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:32.543983 8061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:32.552287 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:32.552356 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:32.552543 8061 server_base.cc:1034] running on GCE node
W20260619 16:46:32.552388 8069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:32.553221 8061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:32.554388 8061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:32.555576 8061 hybrid_clock.cc:648] HybridClock initialized: now 1781887592555510 us; error 78 us; skew 500 ppm
Jun 19 16:46:32 dist-test-slave-cgwx krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887592, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:32.562551 8061 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:32.563398 8061 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:32.563467 8061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:32.563673 8061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:32.565729 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "053891e7f4e349f398c9ebcdfea42347"
format_stamp: "Formatted at 2026-06-19 16:46:32 on dist-test-slave-cgwx"
server_key: "9ff6826505755b53c6297c177c403bcd"
server_key_iv: "bf117786f30e3af385a9ecdb336bcb28"
server_key_version: "encryptionkey@0"
I20260619 16:46:32.566242 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "053891e7f4e349f398c9ebcdfea42347"
format_stamp: "Formatted at 2026-06-19 16:46:32 on dist-test-slave-cgwx"
server_key: "9ff6826505755b53c6297c177c403bcd"
server_key_iv: "bf117786f30e3af385a9ecdb336bcb28"
server_key_version: "encryptionkey@0"
I20260619 16:46:32.569960 8061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:46:32.572368 8075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:32.573601 8061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:32.573699 8061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "053891e7f4e349f398c9ebcdfea42347"
format_stamp: "Formatted at 2026-06-19 16:46:32 on dist-test-slave-cgwx"
server_key: "9ff6826505755b53c6297c177c403bcd"
server_key_iv: "bf117786f30e3af385a9ecdb336bcb28"
server_key_version: "encryptionkey@0"
I20260619 16:46:32.573783 8061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:32.595129 8061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:32.598606 8061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:32.598767 8061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:32.606182 8061 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34943
I20260619 16:46:32.606462 8127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34943 every 8 connection(s)
I20260619 16:46:32.606900 8061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:32.609676 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8061
I20260619 16:46:32.609750 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:32.609828 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:32.610102 4006 external_mini_cluster.cc:1428] Setting key b5dca84f2f5f7179ec03563d566a11e7
I20260619 16:46:32.617230 8128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347: Bootstrap starting.
I20260619 16:46:32.619637 8128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347: Neither blocks nor log segments found. Creating new log.
Jun 19 16:46:32 dist-test-slave-cgwx krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:32.620440 8128 log.cc:824] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:32.623114 8128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347: No bootstrap required, opened a new log
I20260619 16:46:32.626435 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:32.611526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.3.233.190:34943)
0619 16:46:32.611936 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:32.611955 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:32.611997 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:46:32.612795 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:32.612819 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:32.613260 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:32.613646 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:32.614297 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:32.615897 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:32.616896 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:32.617608 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:32.617759 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:32.620572 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:32.620581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:32.620598 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:32.620630 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:32.622745 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:32.623460 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:32.623468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:32.623476 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:32.623621 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:32.623961 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:32.623964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:32.623965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:32.624478 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:32.624671 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:32.624938 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 16:46:32.625247 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":103,"threads_started":1}
I20260619 16:46:32.628994 8128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053891e7f4e349f398c9ebcdfea42347" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34943 } }
I20260619 16:46:32.629278 8128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:32.629345 8128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053891e7f4e349f398c9ebcdfea42347, State: Initialized, Role: FOLLOWER
I20260619 16:46:32.629765 8128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053891e7f4e349f398c9ebcdfea42347" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34943 } }
I20260619 16:46:32.629918 8128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:32.629982 8128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:32.630074 8128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:32.631072 8128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053891e7f4e349f398c9ebcdfea42347" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34943 } }
I20260619 16:46:32.631431 8128 leader_election.cc:302] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 053891e7f4e349f398c9ebcdfea42347; no voters:
I20260619 16:46:32.631771 8128 leader_election.cc:288] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:32.631951 8133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:32.632159 8133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [term 1 LEADER]: Becoming Leader. State: Replica: 053891e7f4e349f398c9ebcdfea42347, State: Running, Role: LEADER
I20260619 16:46:32.632465 8133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053891e7f4e349f398c9ebcdfea42347" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34943 } }
I20260619 16:46:32.633038 8128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:32.634133 8135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 053891e7f4e349f398c9ebcdfea42347. Latest consensus state: current_term: 1 leader_uuid: "053891e7f4e349f398c9ebcdfea42347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053891e7f4e349f398c9ebcdfea42347" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34943 } } }
I20260619 16:46:32.634251 8135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:32.634629 8142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:32.634620 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "053891e7f4e349f398c9ebcdfea42347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053891e7f4e349f398c9ebcdfea42347" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34943 } } }
I20260619 16:46:32.635424 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:32.637924 8142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:32.643221 8142 catalog_manager.cc:1347] Generated new cluster ID: 926027a2a9034103a75a9a680262e6af
I20260619 16:46:32.643303 8142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:32.652240 8142 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:32.653471 8142 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:32.658699 8142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347: Generated new TSK 0
I20260619 16:46:32.659539 8142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:32.690393 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34943
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:32.803336 8154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:32.803643 8154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:32.803725 8154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:32.807977 8154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:32.808063 8154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:32.808156 8154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:32.813639 8154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:32.814771 8154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:32.815879 8154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:32.825053 8162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:32.825100 8159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:32.825282 8154 server_base.cc:1034] running on GCE node
W20260619 16:46:32.825114 8160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:32.825821 8154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:32.826411 8154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:32.827574 8154 hybrid_clock.cc:648] HybridClock initialized: now 1781887592827555 us; error 31 us; skew 500 ppm
Jun 19 16:46:32 dist-test-slave-cgwx krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887592, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:32.830312 8154 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:32.831130 8154 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:32.831192 8154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:32.831424 8154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:32.833611 8154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39"
format_stamp: "Formatted at 2026-06-19 16:46:32 on dist-test-slave-cgwx"
server_key: "22abb68bf8332b112d03ce5e0623999f"
server_key_iv: "e9db780db671a44b7c8cbce461c17f30"
server_key_version: "encryptionkey@0"
I20260619 16:46:32.834249 8154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39"
format_stamp: "Formatted at 2026-06-19 16:46:32 on dist-test-slave-cgwx"
server_key: "22abb68bf8332b112d03ce5e0623999f"
server_key_iv: "e9db780db671a44b7c8cbce461c17f30"
server_key_version: "encryptionkey@0"
I20260619 16:46:32.838110 8154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:46:32.840731 8168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:32.842247 8154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:46:32.842383 8154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39"
format_stamp: "Formatted at 2026-06-19 16:46:32 on dist-test-slave-cgwx"
server_key: "22abb68bf8332b112d03ce5e0623999f"
server_key_iv: "e9db780db671a44b7c8cbce461c17f30"
server_key_version: "encryptionkey@0"
I20260619 16:46:32.842497 8154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:32.856146 8154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:32.859460 8154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:32.859668 8154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:32.860282 8154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:32.861151 8154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:32.861197 8154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:32.861268 8154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:32.861299 8154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:32.870349 8154 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:44113
I20260619 16:46:32.870366 8281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:44113 every 8 connection(s)
I20260619 16:46:32.871042 8154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:32.877034 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8154
I20260619 16:46:32.877156 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:32.877439 4006 external_mini_cluster.cc:1428] Setting key 08819ca1d219013b0729e4742c09b3b5
Jun 19 16:46:32 dist-test-slave-cgwx krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887592, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:32.885360 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:32.872840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:43383 (local address 127.3.233.190:34943)
0619 16:46:32.873001 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:32.873006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:32.873870 (+ 864us) server_negotiation.cc:407] Connection header received
0619 16:46:32.875190 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:32.875193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:32.875241 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:32.875307 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:32.876429 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:32.877173 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:32.878737 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:32.879081 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:32.879209 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:32.881736 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:32.881740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:32.881742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:32.881767 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:32.883318 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:32.883824 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:32.883829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:32.883831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:32.883918 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:32.884267 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:32.884270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:32.884272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:32.884457 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:32.884563 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:32.885066 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 16:46:32.885189 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:32.886180 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:32.873265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34943 (local address 127.3.233.129:43383)
0619 16:46:32.873715 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:32.873759 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:32.874988 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:32.875483 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:32.875489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:32.875835 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:32.876297 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:32.876310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:32.877298 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:32.877300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:32.878619 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:32.878629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:32.879233 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:32.879236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:32.879418 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:32.880103 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:32.880120 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:32.881564 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:32.883435 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:32.883441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:32.883457 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:32.883719 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:32.884016 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:32.884019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:32.884021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:32.884165 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:32.884559 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:32.884567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:32.884825 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 16:46:32.885017 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:46:32.885349 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":98,"threads_started":1}
I20260619 16:46:32.887511 8282 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34943
I20260619 16:46:32.887778 8282 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:32.888236 8282 heartbeater.cc:511] Master 127.3.233.190:34943 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:32.889955 8089 ts_manager.cc:194] Registered new tserver with Master: a2ca9494fc8e4380b8ed5ecc9e1e4b39 (127.3.233.129:44113)
I20260619 16:46:32.891175 8089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:43383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:32.905812 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34943
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:33.022270 8287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:33.022527 8287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:33.022574 8287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:33.026789 8287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:33.026866 8287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:33.026993 8287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:33.032611 8287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:33.033799 8287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:33.034965 8287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:33.044804 8292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:33.044775 8295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:33.044993 8287 server_base.cc:1034] running on GCE node
W20260619 16:46:33.044929 8293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:33.045640 8287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:33.046308 8287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:33.047504 8287 hybrid_clock.cc:648] HybridClock initialized: now 1781887593047474 us; error 44 us; skew 500 ppm
Jun 19 16:46:33 dist-test-slave-cgwx krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:33.055557 8287 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:33.056432 8287 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:33.056502 8287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:33.056732 8287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:33.059000 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "9e02918c907742629f6245d1614b85a8"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "a4207e6a9c71351eedf25e04b36110b2"
server_key_iv: "9ec48a5518a96a688658278532f27ac1"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.059612 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "9e02918c907742629f6245d1614b85a8"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "a4207e6a9c71351eedf25e04b36110b2"
server_key_iv: "9ec48a5518a96a688658278532f27ac1"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.063589 8287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:33.066128 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.067587 8287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:33.067727 8287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "9e02918c907742629f6245d1614b85a8"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "a4207e6a9c71351eedf25e04b36110b2"
server_key_iv: "9ec48a5518a96a688658278532f27ac1"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.067847 8287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:33.090361 8287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:33.093784 8287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:33.093993 8287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:33.094583 8287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:33.095486 8287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:33.095556 8287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.095626 8287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:33.095676 8287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.104485 8287 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:43833
I20260619 16:46:33.104501 8414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:43833 every 8 connection(s)
I20260619 16:46:33.105080 8287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:33.113040 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8287
I20260619 16:46:33.113174 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:33.113517 4006 external_mini_cluster.cc:1428] Setting key 8e0a5440b65b1f34c7d8742e994b3a98
Jun 19 16:46:33 dist-test-slave-cgwx krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:33.119449 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.106830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:56583 (local address 127.3.233.190:34943)
0619 16:46:33.107015 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.107020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.107880 (+ 860us) server_negotiation.cc:407] Connection header received
0619 16:46:33.109129 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.109131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.109178 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.109253 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:33.110474 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.111124 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.112272 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.112479 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.112553 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.115401 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:33.115405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:33.115409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:33.115443 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:33.117110 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:33.117750 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:33.117753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:33.117754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:33.117826 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:33.118199 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:33.118202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:33.118203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:33.118366 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:33.118446 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.119172 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.119302 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:46:33.120663 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.107282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34943 (local address 127.3.233.130:56583)
0619 16:46:33.107726 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.107767 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.108942 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.109471 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.109478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.109856 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:33.110345 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.110358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.111242 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.111244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.112170 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.112179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.112626 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.112629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.112791 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.113437 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:33.113452 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:33.115232 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:33.117246 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:33.117253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:33.117272 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:33.117650 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:33.117917 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:33.117923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:33.117928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:33.118109 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:33.118472 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:33.118481 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:33.118820 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.119062 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.119460 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260619 16:46:33.121906 8415 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34943
I20260619 16:46:33.122167 8415 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:33.122720 8415 heartbeater.cc:511] Master 127.3.233.190:34943 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:46:33.123888 8089 ts_manager.cc:194] Registered new tserver with Master: 9e02918c907742629f6245d1614b85a8 (127.3.233.130:43833)
I20260619 16:46:33.124771 8089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:56583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:33.145722 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34943
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:33.268421 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:33.268680 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:33.268771 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:33.273108 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:33.273167 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:33.273278 8420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:33.278618 8420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:33.279690 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:33.280676 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:33.290201 8426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:33.290201 8428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:33.290248 8425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:33.290382 8420 server_base.cc:1034] running on GCE node
I20260619 16:46:33.291000 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:33.291647 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:33.292804 8420 hybrid_clock.cc:648] HybridClock initialized: now 1781887593292786 us; error 33 us; skew 500 ppm
Jun 19 16:46:33 dist-test-slave-cgwx krb5kdc[8048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:33.295598 8420 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:33.296298 8420 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:33.296360 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:33.296521 8420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:33.298547 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "d6dee14a7cab4740be9f2f2203be45b8"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "d03d848b931929b3d6abdc36cc0cb8db"
server_key_iv: "218a35b53b1fdf8cb8e43e73d01b7a77"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.299120 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "d6dee14a7cab4740be9f2f2203be45b8"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "d03d848b931929b3d6abdc36cc0cb8db"
server_key_iv: "218a35b53b1fdf8cb8e43e73d01b7a77"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.302754 8420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:33.305253 8434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.306610 8420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:33.306757 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "d6dee14a7cab4740be9f2f2203be45b8"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "d03d848b931929b3d6abdc36cc0cb8db"
server_key_iv: "218a35b53b1fdf8cb8e43e73d01b7a77"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.306876 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:33.323091 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:33.326200 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:33.326402 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:33.327019 8420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:33.327833 8420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:33.327916 8420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.327992 8420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:33.328034 8420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.336812 8420 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:35169
I20260619 16:46:33.336858 8547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:35169 every 8 connection(s)
I20260619 16:46:33.337443 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:33.343137 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8420
I20260619 16:46:33.343283 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:33.343587 4006 external_mini_cluster.cc:1428] Setting key fa17aea1b9330399fc81f61ce62692f1
Jun 19 16:46:33 dist-test-slave-cgwx krb5kdc[8048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:33.351779 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.339290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:40339 (local address 127.3.233.190:34943)
0619 16:46:33.339428 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.339432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.340366 (+ 934us) server_negotiation.cc:407] Connection header received
0619 16:46:33.341713 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.341716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.341792 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.341940 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:33.343326 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.343942 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.345207 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.345431 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.345559 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.348094 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:33.348097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:33.348099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:33.348122 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:33.349489 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:33.349985 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:33.349988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:33.349989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:33.350060 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:33.350396 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:33.350400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:33.350401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:33.350599 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:33.350701 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.351496 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.351617 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:33.352669 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.339700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34943 (local address 127.3.233.131:40339)
0619 16:46:33.340189 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.340246 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.341504 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.342116 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.342123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.342481 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:33.343138 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.343162 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.344068 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.344078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.345078 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.345087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.345593 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.345597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.345770 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.346410 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:33.346426 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:33.347941 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:33.349597 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:33.349602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:33.349615 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:33.349875 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:33.350137 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:33.350139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:33.350141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:33.350272 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:33.350708 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:33.350714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:33.351231 (+ 517us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.351453 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.351804 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260619 16:46:33.353830 8548 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34943
I20260619 16:46:33.354106 8548 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:33.354549 8548 heartbeater.cc:511] Master 127.3.233.190:34943 requested a full tablet report, sending...
I20260619 16:46:33.355581 8089 ts_manager.cc:194] Registered new tserver with Master: d6dee14a7cab4740be9f2f2203be45b8 (127.3.233.131:35169)
I20260619 16:46:33.356324 8089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:40339
I20260619 16:46:33.357482 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:33.366359 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.359453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.3.233.190:34943)
0619 16:46:33.359571 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.359574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.359841 (+ 267us) server_negotiation.cc:407] Connection header received
0619 16:46:33.359934 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.359936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.359973 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.360037 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:33.360663 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.361214 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.361977 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.362177 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.362252 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.363609 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:33.363612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:33.363614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:33.363632 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:33.364951 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:33.365332 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:33.365335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:33.365336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:33.365407 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:33.365624 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:33.365626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:33.365627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:33.365869 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:33.365973 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.366157 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.366247 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:46:33.369208 8089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:33.372318 8089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:33.386503 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.380605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:43833 (local address 127.0.0.1:54702)
0619 16:46:33.380937 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.380989 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.381113 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.382001 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.382006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.382025 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:33.382149 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.382154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.383606 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.383614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.384781 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.384790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.385903 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.385906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.386024 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.386070 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.386180 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.386303 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":25,"thread_start_us":62,"threads_started":1}
I20260619 16:46:33.386765 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.380681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44113 (local address 127.0.0.1:54594)
0619 16:46:33.381388 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.381402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.381490 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.382382 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.382385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.382405 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:33.382508 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.382513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.384179 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.384183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.385300 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.385309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.386228 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.386232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.386343 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.386369 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.386440 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.386508 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260619 16:46:33.386847 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.380703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.3.233.130:43833)
0619 16:46:33.381486 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.381500 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.381524 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:46:33.381601 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.381605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.381831 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.382325 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:33.382490 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.383449 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.384918 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.385776 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.385880 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.386021 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.386591 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.386694 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":167,"threads_started":1}
I20260619 16:46:33.386972 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.381400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.3.233.129:44113)
0619 16:46:33.381921 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.381928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.381950 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:46:33.382019 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.382024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.382212 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.382645 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:33.382823 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.384035 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.385421 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.386090 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.386202 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.386326 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.386488 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.386596 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":92,"threads_started":1}
I20260619 16:46:33.388110 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.381408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:35169 (local address 127.0.0.1:37150)
0619 16:46:33.381815 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.381828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.381914 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.382879 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.382882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.382900 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:33.383002 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.383008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.384287 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.384290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.385377 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.385390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.386565 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.386568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.387828 (+ 1260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.387852 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.387927 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.387990 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":26,"thread_start_us":83,"threads_started":1}
I20260619 16:46:33.388553 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.381800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.3.233.131:35169)
0619 16:46:33.382356 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.382368 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.382388 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:46:33.382453 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.382458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.382731 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.382899 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:33.383095 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.384164 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.385503 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.386275 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.386372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.386507 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.388320 (+ 1813us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.388411 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":101,"threads_started":1}
I20260619 16:46:33.389631 8216 tablet_service.cc:1467] Processing CreateTablet for tablet 1377aeb4282b473e911525b103056068 (DEFAULT_TABLE table=test-table [id=eb82014b5ab040caacf8b7f7445d3f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:33.389635 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 1377aeb4282b473e911525b103056068 (DEFAULT_TABLE table=test-table [id=eb82014b5ab040caacf8b7f7445d3f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:33.390638 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1377aeb4282b473e911525b103056068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:33.390671 8216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1377aeb4282b473e911525b103056068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:33.391074 8482 tablet_service.cc:1467] Processing CreateTablet for tablet 1377aeb4282b473e911525b103056068 (DEFAULT_TABLE table=test-table [id=eb82014b5ab040caacf8b7f7445d3f99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:33.391975 8482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1377aeb4282b473e911525b103056068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:33.396646 8565 tablet_bootstrap.cc:492] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Bootstrap starting.
I20260619 16:46:33.398294 8566 tablet_bootstrap.cc:492] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39: Bootstrap starting.
I20260619 16:46:33.399191 8565 tablet_bootstrap.cc:654] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:33.399268 8567 tablet_bootstrap.cc:492] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Bootstrap starting.
I20260619 16:46:33.400305 8565 log.cc:824] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:33.400481 8566 tablet_bootstrap.cc:654] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:33.401275 8566 log.cc:824] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:33.401482 8567 tablet_bootstrap.cc:654] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:33.402199 8565 tablet_bootstrap.cc:492] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: No bootstrap required, opened a new log
I20260619 16:46:33.402401 8565 ts_tablet_manager.cc:1397] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:46:33.402701 8567 log.cc:824] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:33.403254 8566 tablet_bootstrap.cc:492] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39: No bootstrap required, opened a new log
I20260619 16:46:33.403389 8566 ts_tablet_manager.cc:1397] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:46:33.405545 8567 tablet_bootstrap.cc:492] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: No bootstrap required, opened a new log
I20260619 16:46:33.405718 8567 ts_tablet_manager.cc:1397] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:46:33.405973 8565 raft_consensus.cc:348] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.406257 8565 raft_consensus.cc:374] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:33.406315 8565 raft_consensus.cc:729] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e02918c907742629f6245d1614b85a8, State: Initialized, Role: FOLLOWER
I20260619 16:46:33.406734 8565 consensus_queue.cc:260] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.407394 8415 heartbeater.cc:503] Master 127.3.233.190:34943 was elected leader, sending a full tablet report...
I20260619 16:46:33.407431 8566 raft_consensus.cc:348] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.407668 8566 raft_consensus.cc:374] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:33.407721 8566 raft_consensus.cc:729] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ca9494fc8e4380b8ed5ecc9e1e4b39, State: Initialized, Role: FOLLOWER
I20260619 16:46:33.407727 8565 ts_tablet_manager.cc:1428] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:46:33.408123 8566 consensus_queue.cc:260] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.408771 8282 heartbeater.cc:503] Master 127.3.233.190:34943 was elected leader, sending a full tablet report...
I20260619 16:46:33.409085 8566 ts_tablet_manager.cc:1428] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:46:33.409426 8567 raft_consensus.cc:348] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.409662 8567 raft_consensus.cc:374] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:33.409718 8567 raft_consensus.cc:729] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dee14a7cab4740be9f2f2203be45b8, State: Initialized, Role: FOLLOWER
I20260619 16:46:33.410035 8567 consensus_queue.cc:260] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.410668 8548 heartbeater.cc:503] Master 127.3.233.190:34943 was elected leader, sending a full tablet report...
I20260619 16:46:33.410902 8567 ts_tablet_manager.cc:1428] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:46:33.446319 8573 raft_consensus.cc:482] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:33.446542 8573 raft_consensus.cc:504] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.447731 8573 leader_election.cc:288] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e02918c907742629f6245d1614b85a8 (127.3.233.130:43833), a2ca9494fc8e4380b8ed5ecc9e1e4b39 (127.3.233.129:44113)
I20260619 16:46:33.451877 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.448111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:50795 (local address 127.3.233.129:44113)
0619 16:46:33.448233 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.448238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.448332 (+ 94us) server_negotiation.cc:407] Connection header received
0619 16:46:33.448481 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.448486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.448550 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.448632 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:33.448974 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.449743 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.450807 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.451314 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.451452 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.451507 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.451679 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.451751 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:46:33.451877 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.448024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44113 (local address 127.3.233.131:50795)
0619 16:46:33.448229 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.448245 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.448366 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.448676 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.448682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.448705 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:33.448841 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.448851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.449850 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.449852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.450635 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.450642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.451443 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.451446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.451557 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.451575 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.451664 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.451736 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 16:46:33.452031 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.448137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:46629 (local address 127.3.233.130:43833)
0619 16:46:33.448241 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.448246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.448648 (+ 402us) server_negotiation.cc:407] Connection header received
0619 16:46:33.448899 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.448902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.448973 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.449048 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:33.449329 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.449983 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.451038 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.451439 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.451577 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.451627 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.451816 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.451887 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:46:33.452282 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.448024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:43833 (local address 127.3.233.131:46629)
0619 16:46:33.448477 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.448490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.448602 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.449099 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.449102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.449118 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:33.449211 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.449217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.450093 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.450097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.450871 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:33.450879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.451571 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.451575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:33.451676 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.451697 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.451793 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.451861 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":34,"thread_start_us":125,"threads_started":1}
I20260619 16:46:33.452735 8236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1377aeb4282b473e911525b103056068" candidate_uuid: "d6dee14a7cab4740be9f2f2203be45b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" is_pre_election: true
I20260619 16:46:33.452943 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1377aeb4282b473e911525b103056068" candidate_uuid: "d6dee14a7cab4740be9f2f2203be45b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e02918c907742629f6245d1614b85a8" is_pre_election: true
I20260619 16:46:33.453074 8236 raft_consensus.cc:2454] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6dee14a7cab4740be9f2f2203be45b8 in term 0.
I20260619 16:46:33.453266 8369 raft_consensus.cc:2454] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6dee14a7cab4740be9f2f2203be45b8 in term 0.
I20260619 16:46:33.453614 8436 leader_election.cc:302] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2ca9494fc8e4380b8ed5ecc9e1e4b39, d6dee14a7cab4740be9f2f2203be45b8; no voters:
I20260619 16:46:33.453941 8573 raft_consensus.cc:2781] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:33.454016 8573 raft_consensus.cc:482] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:33.454047 8573 raft_consensus.cc:3037] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:33.455197 8573 raft_consensus.cc:504] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.455549 8573 leader_election.cc:288] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [CANDIDATE]: Term 1 election: Requested vote from peers 9e02918c907742629f6245d1614b85a8 (127.3.233.130:43833), a2ca9494fc8e4380b8ed5ecc9e1e4b39 (127.3.233.129:44113)
I20260619 16:46:33.455986 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1377aeb4282b473e911525b103056068" candidate_uuid: "d6dee14a7cab4740be9f2f2203be45b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e02918c907742629f6245d1614b85a8"
I20260619 16:46:33.455996 8236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1377aeb4282b473e911525b103056068" candidate_uuid: "d6dee14a7cab4740be9f2f2203be45b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39"
I20260619 16:46:33.456118 8369 raft_consensus.cc:3037] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:33.456118 8236 raft_consensus.cc:3037] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:33.457442 8369 raft_consensus.cc:2454] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6dee14a7cab4740be9f2f2203be45b8 in term 1.
I20260619 16:46:33.457442 8236 raft_consensus.cc:2454] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6dee14a7cab4740be9f2f2203be45b8 in term 1.
I20260619 16:46:33.457798 8436 leader_election.cc:302] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2ca9494fc8e4380b8ed5ecc9e1e4b39, d6dee14a7cab4740be9f2f2203be45b8; no voters:
I20260619 16:46:33.457973 8573 raft_consensus.cc:2781] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:33.458225 8573 raft_consensus.cc:686] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 1 LEADER]: Becoming Leader. State: Replica: d6dee14a7cab4740be9f2f2203be45b8, State: Running, Role: LEADER
I20260619 16:46:33.458628 8573 consensus_queue.cc:237] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } }
I20260619 16:46:33.462110 8088 catalog_manager.cc:5495] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6dee14a7cab4740be9f2f2203be45b8 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "d6dee14a7cab4740be9f2f2203be45b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6dee14a7cab4740be9f2f2203be45b8" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 35169 } health_report { overall_health: HEALTHY } } }
I20260619 16:46:33.487397 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.483778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.3.233.131:35169)
0619 16:46:33.483914 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.483919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.483935 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:33.484027 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.484029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.484082 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.484194 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:33.484444 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.484952 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.485860 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:33.486069 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:33.486183 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:33.486494 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:33.486906 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:33.487110 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.487172 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.487227 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:33.493196 8236 raft_consensus.cc:1261] T 1377aeb4282b473e911525b103056068 P a2ca9494fc8e4380b8ed5ecc9e1e4b39 [term 1 FOLLOWER]: Refusing update from remote peer d6dee14a7cab4740be9f2f2203be45b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:33.493196 8369 raft_consensus.cc:1261] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 1 FOLLOWER]: Refusing update from remote peer d6dee14a7cab4740be9f2f2203be45b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:33.493980 8575 consensus_queue.cc:1035] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e02918c907742629f6245d1614b85a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:33.494176 8573 consensus_queue.cc:1035] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2ca9494fc8e4380b8ed5ecc9e1e4b39" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:33.501317 8578 mvcc.cc:205] Tried to move back new op lower bound from 7298611582940385280 to 7298611582810267648. Current Snapshot: MvccSnapshot[applied={T|T < 7298611582940385280}]
I20260619 16:46:33.509783 8088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:33.509989 8088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:33.512598 8088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 053891e7f4e349f398c9ebcdfea42347: Sending DeleteTablet for 3 replicas of tablet 1377aeb4282b473e911525b103056068
I20260619 16:46:33.513427 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 1377aeb4282b473e911525b103056068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:33 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:54702
I20260619 16:46:33.513554 8482 tablet_service.cc:1514] Processing DeleteTablet for tablet 1377aeb4282b473e911525b103056068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:33 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:37150
I20260619 16:46:33.514008 8588 tablet_replica.cc:330] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: stopping tablet replica
I20260619 16:46:33.514027 8216 tablet_service.cc:1514] Processing DeleteTablet for tablet 1377aeb4282b473e911525b103056068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:33 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:54594
I20260619 16:46:33.514134 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8154
I20260619 16:46:33.514362 8587 tablet_replica.cc:330] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: stopping tablet replica
I20260619 16:46:33.515141 8587 raft_consensus.cc:2229] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:33.515237 8588 raft_consensus.cc:2229] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:33.515614 8588 raft_consensus.cc:2258] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:33.515614 8587 raft_consensus.cc:2258] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:33.517036 8588 ts_tablet_manager.cc:1905] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:33.517249 8587 ts_tablet_manager.cc:1905] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:33.519889 8588 ts_tablet_manager.cc:1918] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:33.519992 8588 log.cc:1196] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/1377aeb4282b473e911525b103056068
I20260619 16:46:33.520360 8588 ts_tablet_manager.cc:1939] T 1377aeb4282b473e911525b103056068 P d6dee14a7cab4740be9f2f2203be45b8: Deleting consensus metadata
I20260619 16:46:33.520743 8587 ts_tablet_manager.cc:1918] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:33.520907 8587 log.cc:1196] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/1377aeb4282b473e911525b103056068
I20260619 16:46:33.521291 8587 ts_tablet_manager.cc:1939] T 1377aeb4282b473e911525b103056068 P 9e02918c907742629f6245d1614b85a8: Deleting consensus metadata
I20260619 16:46:33.521549 8077 catalog_manager.cc:4841] TS d6dee14a7cab4740be9f2f2203be45b8 (127.3.233.131:35169): tablet 1377aeb4282b473e911525b103056068 (table test-table [id=eb82014b5ab040caacf8b7f7445d3f99]) successfully deleted
I20260619 16:46:33.522620 8076 catalog_manager.cc:4841] TS 9e02918c907742629f6245d1614b85a8 (127.3.233.130:43833): tablet 1377aeb4282b473e911525b103056068 (table test-table [id=eb82014b5ab040caacf8b7f7445d3f99]) successfully deleted
W20260619 16:46:33.523588 8077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:44113 (error 108)
I20260619 16:46:33.524438 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8287
I20260619 16:46:33.524626 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.524342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44113 (local address 127.0.0.1:54604)
0619 16:46:33.524459 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.524529 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:44113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 16:46:33.525525 8077 catalog_manager.cc:4584] TS a2ca9494fc8e4380b8ed5ecc9e1e4b39 (127.3.233.129:44113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1377aeb4282b473e911525b103056068: Network error: Client connection negotiation failed: client connection to 127.3.233.129:44113: connect: Connection refused (error 111)
I20260619 16:46:33.530881 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8420
I20260619 16:46:33.536628 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8061
2026-06-19T16:46:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2287 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T16:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:33Z Disabled control of system clock
I20260619 16:46:33.567415 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35777
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:46:33.679270 8593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:33.679539 8593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:33.679576 8593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:33.683748 8593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:33.683807 8593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:33.683822 8593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:33.683840 8593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:33.683856 8593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:33.689432 8593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:35777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:33.690652 8593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:33.692325 8593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:33.701069 8599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:33.701071 8601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:33.701097 8598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:33.701424 8593 server_base.cc:1034] running on GCE node
I20260619 16:46:33.701911 8593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:33.703094 8593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:33.704253 8593 hybrid_clock.cc:648] HybridClock initialized: now 1781887593704238 us; error 30 us; skew 500 ppm
I20260619 16:46:33.706053 8593 webserver.cc:460] Webserver started at http://127.3.233.190:34781/ using document root <none> and password file <none>
I20260619 16:46:33.706575 8593 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:33.706633 8593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:33.706791 8593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:33.708869 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "13da73b472a94ad6b2fd55a99de50f79"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "308207d782ffadff6d29eafdb538f829"
server_key_iv: "90782af9ca888cb9dcd89555b8df060c"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.709441 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "13da73b472a94ad6b2fd55a99de50f79"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "308207d782ffadff6d29eafdb538f829"
server_key_iv: "90782af9ca888cb9dcd89555b8df060c"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.713239 8593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:33.715756 8607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.717018 8593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:33.717163 8593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "13da73b472a94ad6b2fd55a99de50f79"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "308207d782ffadff6d29eafdb538f829"
server_key_iv: "90782af9ca888cb9dcd89555b8df060c"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.717278 8593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:33.721940 8593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:33.722633 8593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:33.722826 8593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:33.730023 8593 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:35777
I20260619 16:46:33.730053 8659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:35777 every 8 connection(s)
I20260619 16:46:33.731081 8593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:33.733060 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8593
I20260619 16:46:33.733156 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:33.733512 4006 external_mini_cluster.cc:1428] Setting key 1aa82dfda8d587d54703c0d79f12d203
I20260619 16:46:33.734887 8660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:33.740228 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.734981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.3.233.190:35777)
0619 16:46:33.735763 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.735785 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.735833 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:46:33.736776 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.736802 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.737295 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.737751 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:33.738552 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:33.738559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:33.738576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:33.738588 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:33.738653 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:33.738676 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:33.738834 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.738910 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.739226 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":144,"threads_started":1}
I20260619 16:46:33.742585 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79: Bootstrap starting.
I20260619 16:46:33.745177 8660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:33.745983 8660 log.cc:824] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:33.748241 8660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79: No bootstrap required, opened a new log
I20260619 16:46:33.752153 8660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da73b472a94ad6b2fd55a99de50f79" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35777 } }
I20260619 16:46:33.752414 8660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:33.752498 8660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13da73b472a94ad6b2fd55a99de50f79, State: Initialized, Role: FOLLOWER
I20260619 16:46:33.752940 8660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da73b472a94ad6b2fd55a99de50f79" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35777 } }
I20260619 16:46:33.753065 8660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:33.753161 8660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:33.753333 8660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:33.754347 8660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da73b472a94ad6b2fd55a99de50f79" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35777 } }
I20260619 16:46:33.754724 8660 leader_election.cc:302] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13da73b472a94ad6b2fd55a99de50f79; no voters:
I20260619 16:46:33.755089 8660 leader_election.cc:288] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:33.755218 8665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:33.755488 8665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [term 1 LEADER]: Becoming Leader. State: Replica: 13da73b472a94ad6b2fd55a99de50f79, State: Running, Role: LEADER
I20260619 16:46:33.755820 8665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da73b472a94ad6b2fd55a99de50f79" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35777 } }
I20260619 16:46:33.756511 8660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:33.757688 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13da73b472a94ad6b2fd55a99de50f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da73b472a94ad6b2fd55a99de50f79" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35777 } } }
I20260619 16:46:33.757844 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:33.758101 8667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13da73b472a94ad6b2fd55a99de50f79. Latest consensus state: current_term: 1 leader_uuid: "13da73b472a94ad6b2fd55a99de50f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da73b472a94ad6b2fd55a99de50f79" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35777 } } }
I20260619 16:46:33.758185 8667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:33.759227 8674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:33.761858 8674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:33.767669 8674 catalog_manager.cc:1347] Generated new cluster ID: 3811d5f94c944f5e94ae9acc71cb3934
I20260619 16:46:33.767747 8674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:33.786831 8674 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:33.787199 8674 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:33.793191 8674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79: Generated new TSK 0
I20260619 16:46:33.793933 8674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:33.798457 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35777
--builtin_ntp_servers=127.3.233.148:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:46:33.909693 8684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:33.909960 8684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:33.910001 8684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:33.914224 8684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:33.914287 8684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:33.914377 8684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:33.919669 8684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:33.920848 8684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:33.922417 8684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:33.932081 8689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:33.932094 8692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:33.932206 8684 server_base.cc:1034] running on GCE node
W20260619 16:46:33.932472 8690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:33.932859 8684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:33.933554 8684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:33.934743 8684 hybrid_clock.cc:648] HybridClock initialized: now 1781887593934712 us; error 50 us; skew 500 ppm
I20260619 16:46:33.936673 8684 webserver.cc:460] Webserver started at http://127.3.233.129:46803/ using document root <none> and password file <none>
I20260619 16:46:33.937258 8684 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:33.937361 8684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:33.937592 8684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:33.939772 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "7629cdc4f54748038af3ecc53450517c"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "0f95024412fe839aa79b2fbea21e1850"
server_key_iv: "cd3b43a99251e0c3509a469ca0e09975"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.940414 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "7629cdc4f54748038af3ecc53450517c"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "0f95024412fe839aa79b2fbea21e1850"
server_key_iv: "cd3b43a99251e0c3509a469ca0e09975"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.944518 8684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:33.947144 8698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.948589 8684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:33.948762 8684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "7629cdc4f54748038af3ecc53450517c"
format_stamp: "Formatted at 2026-06-19 16:46:33 on dist-test-slave-cgwx"
server_key: "0f95024412fe839aa79b2fbea21e1850"
server_key_iv: "cd3b43a99251e0c3509a469ca0e09975"
server_key_version: "encryptionkey@0"
I20260619 16:46:33.948921 8684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:33.954157 8684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:33.954872 8684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:33.955097 8684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:33.955681 8684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:33.956729 8684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:33.956811 8684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.956889 8684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:33.956949 8684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:33.966480 8684 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:41283
I20260619 16:46:33.966540 8811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:41283 every 8 connection(s)
I20260619 16:46:33.967573 8684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:33.973639 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.969446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:57433 (local address 127.3.233.190:35777)
0619 16:46:33.969626 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:33.969631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:33.970771 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 16:46:33.972045 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:33.972047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:33.972085 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:33.972168 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:33.973240 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:33.973242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:33.973244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:33.973256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:33.973275 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:33.973292 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:33.973375 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:33.973491 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:46:33.973529 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:46:33.974454 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:33.969955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35777 (local address 127.3.233.129:57433)
0619 16:46:33.970608 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:33.970649 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:33.971847 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:33.972319 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:33.972325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:33.972411 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:33.972995 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:33.972997 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:33.973036 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:33.973102 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:33.973373 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:33.973379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:33.973383 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:46:33.973508 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 16:46:33.973683 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260619 16:46:33.975203 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8684
I20260619 16:46:33.975374 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:33.975697 4006 external_mini_cluster.cc:1428] Setting key 25bf286e38d4a9b08db105948834327a
I20260619 16:46:33.975865 8812 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35777
I20260619 16:46:33.976089 8812 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:33.976586 8812 heartbeater.cc:511] Master 127.3.233.190:35777 requested a full tablet report, sending...
I20260619 16:46:33.977840 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35777
--builtin_ntp_servers=127.3.233.148:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:46:33.978259 8624 ts_manager.cc:194] Registered new tserver with Master: 7629cdc4f54748038af3ecc53450517c (127.3.233.129:41283)
W20260619 16:46:34.095036 8815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:34.095281 8815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:34.095320 8815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:34.099287 8815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:34.099335 8815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:34.099390 8815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:34.104131 8815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:34.105118 8815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:34.106412 8815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:34.115938 8821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:34.115983 8823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.116142 8815 server_base.cc:1034] running on GCE node
W20260619 16:46:34.115926 8820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.116681 8815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:34.117292 8815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:34.118479 8815 hybrid_clock.cc:648] HybridClock initialized: now 1781887594118461 us; error 29 us; skew 500 ppm
I20260619 16:46:34.120424 8815 webserver.cc:460] Webserver started at http://127.3.233.130:38865/ using document root <none> and password file <none>
I20260619 16:46:34.121025 8815 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:34.121086 8815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:34.121328 8815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:34.123416 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "25a4cccf0ee7483cb1dda42b52696b64"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "0663c0315c54eadda508de74102362bc"
server_key_iv: "d4d1376a168583db28f671872fe747bd"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.123962 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "25a4cccf0ee7483cb1dda42b52696b64"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "0663c0315c54eadda508de74102362bc"
server_key_iv: "d4d1376a168583db28f671872fe747bd"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.127552 8815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:46:34.130101 8829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.131407 8815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:34.131533 8815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "25a4cccf0ee7483cb1dda42b52696b64"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "0663c0315c54eadda508de74102362bc"
server_key_iv: "d4d1376a168583db28f671872fe747bd"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.131649 8815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:34.136149 8815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:34.136735 8815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:34.136911 8815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:34.137441 8815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:34.138342 8815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:34.138413 8815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.138484 8815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:34.138536 8815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.147460 8815 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:42961
I20260619 16:46:34.147481 8942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:42961 every 8 connection(s)
I20260619 16:46:34.148409 8815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:34.154623 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.150268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:43295 (local address 127.3.233.190:35777)
0619 16:46:34.150413 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.150417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.151565 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 16:46:34.152890 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.152894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.152936 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.153016 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.154127 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.154131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.154134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.154146 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.154169 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.154185 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.154290 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.154438 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.154483 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:46:34.155085 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8815
I20260619 16:46:34.155213 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:34.155499 4006 external_mini_cluster.cc:1428] Setting key 2c49ea1b767ec0f78f22f45e3a094896
I20260619 16:46:34.155686 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.150696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35777 (local address 127.3.233.130:43295)
0619 16:46:34.151413 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.151453 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.152672 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.153197 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.153204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.153346 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:34.153861 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:34.153863 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:34.153904 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:34.153973 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:34.154282 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:34.154290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:34.154297 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:46:34.154482 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:46:34.154721 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":106,"threads_started":1}
I20260619 16:46:34.156948 8943 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35777
I20260619 16:46:34.157191 8943 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:34.157531 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35777
--builtin_ntp_servers=127.3.233.148:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:46:34.157617 8943 heartbeater.cc:511] Master 127.3.233.190:35777 requested a full tablet report, sending...
I20260619 16:46:34.158821 8624 ts_manager.cc:194] Registered new tserver with Master: 25a4cccf0ee7483cb1dda42b52696b64 (127.3.233.130:42961)
W20260619 16:46:34.270222 8946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:34.270542 8946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:34.270633 8946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:34.274787 8946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:34.274875 8946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:34.275009 8946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:34.279798 8946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:34.280900 8946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:34.282370 8946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:34.292217 8952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:34.292248 8951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.292529 8946 server_base.cc:1034] running on GCE node
W20260619 16:46:34.292272 8954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.292969 8946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:34.293674 8946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:34.294905 8946 hybrid_clock.cc:648] HybridClock initialized: now 1781887594294887 us; error 31 us; skew 500 ppm
I20260619 16:46:34.296875 8946 webserver.cc:460] Webserver started at http://127.3.233.131:46525/ using document root <none> and password file <none>
I20260619 16:46:34.297492 8946 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:34.297588 8946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:34.297821 8946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:34.299919 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "158a55b7e76d4443935a327ff085e33d"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "4fa76b651763fc8a3e89a188f8c82b6e"
server_key_iv: "867137b34cd5fc27323c10f9e4334755"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.300472 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "158a55b7e76d4443935a327ff085e33d"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "4fa76b651763fc8a3e89a188f8c82b6e"
server_key_iv: "867137b34cd5fc27323c10f9e4334755"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.304105 8946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:46:34.306377 8960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.307789 8946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:46:34.307922 8946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "158a55b7e76d4443935a327ff085e33d"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "4fa76b651763fc8a3e89a188f8c82b6e"
server_key_iv: "867137b34cd5fc27323c10f9e4334755"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.308028 8946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:34.312434 8946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:34.313005 8946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:34.313176 8946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:34.313752 8946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:34.314601 8946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:34.314670 8946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.314764 8946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:34.314814 8946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.323367 8946 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:45695
I20260619 16:46:34.323444 9073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:45695 every 8 connection(s)
I20260619 16:46:34.324280 8946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:34.330309 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.326062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:52157 (local address 127.3.233.190:35777)
0619 16:46:34.326202 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.326207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.327377 (+ 1170us) server_negotiation.cc:407] Connection header received
0619 16:46:34.328678 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.328681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.328735 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.328845 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.329859 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.329862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.329865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.329877 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.329904 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.329912 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.330008 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.330133 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.330176 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:46:34.331161 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.326481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35777 (local address 127.3.233.131:52157)
0619 16:46:34.327213 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.327253 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.328467 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.329012 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.329018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.329103 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:34.329617 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:34.329618 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:34.329654 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:34.329717 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:34.329992 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:34.329999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:34.330005 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:46:34.330174 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 16:46:34.330431 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260619 16:46:34.332211 9074 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35777
I20260619 16:46:34.332458 9074 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:34.332839 9074 heartbeater.cc:511] Master 127.3.233.190:35777 requested a full tablet report, sending...
I20260619 16:46:34.333817 8624 ts_manager.cc:194] Registered new tserver with Master: 158a55b7e76d4443935a327ff085e33d (127.3.233.131:45695)
I20260619 16:46:34.333877 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 8946
I20260619 16:46:34.333999 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:34.334350 4006 external_mini_cluster.cc:1428] Setting key 658d414f3d49d6a014a38ba2d2e20144
I20260619 16:46:34.336514 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:34.340248 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.338366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.3.233.190:35777)
0619 16:46:34.338490 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.338493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.338726 (+ 233us) server_negotiation.cc:407] Connection header received
0619 16:46:34.338806 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.338809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.338870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.339011 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.339670 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.339673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.339675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.339683 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.339700 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.339706 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.339776 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.340104 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.340143 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:46:34.343299 8624 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:34.346439 8624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:34.358436 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.355600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:41283 (local address 127.0.0.1:55472)
0619 16:46:34.356047 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.356061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.356140 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.357032 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.357035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.357098 (+ 63us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:34.357712 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:34.357714 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:34.357724 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:34.357751 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:34.358067 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:34.358077 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:34.358082 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:46:34.358224 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:46:34.358291 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":76,"threads_started":1}
I20260619 16:46:34.358608 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.355192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:45695 (local address 127.0.0.1:34858)
0619 16:46:34.355588 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.355639 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.355742 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.356881 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.356886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.356965 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:34.357574 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:34.357578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:34.357602 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:34.357679 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:34.358267 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:34.358270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:34.358273 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:46:34.358407 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:46:34.358445 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260619 16:46:34.358687 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.355331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.3.233.131:45695)
0619 16:46:34.356414 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.356431 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.356459 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:46:34.356514 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.356520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.356728 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.356897 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.358079 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.358085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.358095 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.358108 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.358141 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.358162 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.358291 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.358423 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.358529 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":185,"threads_started":1}
I20260619 16:46:34.358487 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.355980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.3.233.129:41283)
0619 16:46:34.356678 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.356696 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.356722 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:46:34.356765 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.356769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.356906 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.357048 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.357870 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.357877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.357889 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.357900 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.357946 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.357981 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.358087 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.358254 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.358332 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":128,"threads_started":1}
I20260619 16:46:34.360090 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.355982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:42961 (local address 127.0.0.1:54986)
0619 16:46:34.357877 (+ 1895us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.357889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.357988 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.359033 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.359036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.359050 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:34.359506 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:34.359508 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:34.359517 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:34.359542 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:34.359855 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:34.359865 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:34.359869 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:46:34.359938 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:46:34.359974 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1830,"thread_start_us":104,"threads_started":1}
I20260619 16:46:34.360179 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.356207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.3.233.130:42961)
0619 16:46:34.356534 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.356544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.357984 (+ 1440us) server_negotiation.cc:407] Connection header received
0619 16:46:34.358645 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.358653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.358870 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.359029 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.359673 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.359679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.359688 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.359698 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.359731 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.359753 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.359857 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.359961 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.360026 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":128,"threads_started":1}
I20260619 16:46:34.361712 8746 tablet_service.cc:1467] Processing CreateTablet for tablet dc324de75a9243beba8adf9239e400e7 (DEFAULT_TABLE table=test-table [id=5c374c4c9d504d88b1182ae83bf942b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:34.361716 9008 tablet_service.cc:1467] Processing CreateTablet for tablet dc324de75a9243beba8adf9239e400e7 (DEFAULT_TABLE table=test-table [id=5c374c4c9d504d88b1182ae83bf942b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:34.362576 8746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc324de75a9243beba8adf9239e400e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:34.362576 9008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc324de75a9243beba8adf9239e400e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:34.365705 8877 tablet_service.cc:1467] Processing CreateTablet for tablet dc324de75a9243beba8adf9239e400e7 (DEFAULT_TABLE table=test-table [id=5c374c4c9d504d88b1182ae83bf942b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:34.366772 8877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc324de75a9243beba8adf9239e400e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:34.368734 9091 tablet_bootstrap.cc:492] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c: Bootstrap starting.
I20260619 16:46:34.370144 9092 tablet_bootstrap.cc:492] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Bootstrap starting.
I20260619 16:46:34.371593 9091 tablet_bootstrap.cc:654] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:34.372182 9092 tablet_bootstrap.cc:654] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:34.372462 9091 log.cc:824] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:34.373003 9092 log.cc:824] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:34.373263 9094 tablet_bootstrap.cc:492] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Bootstrap starting.
I20260619 16:46:34.374594 9091 tablet_bootstrap.cc:492] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c: No bootstrap required, opened a new log
I20260619 16:46:34.374780 9091 ts_tablet_manager.cc:1397] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:46:34.374897 9092 tablet_bootstrap.cc:492] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: No bootstrap required, opened a new log
I20260619 16:46:34.375078 9092 ts_tablet_manager.cc:1397] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260619 16:46:34.375787 9094 tablet_bootstrap.cc:654] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:34.376425 9094 log.cc:824] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:34.377918 9094 tablet_bootstrap.cc:492] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: No bootstrap required, opened a new log
I20260619 16:46:34.378104 9094 ts_tablet_manager.cc:1397] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:46:34.379335 9092 raft_consensus.cc:348] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.379329 9091 raft_consensus.cc:348] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.379592 9091 raft_consensus.cc:374] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:34.379592 9092 raft_consensus.cc:374] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:34.379650 9092 raft_consensus.cc:729] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158a55b7e76d4443935a327ff085e33d, State: Initialized, Role: FOLLOWER
I20260619 16:46:34.379657 9091 raft_consensus.cc:729] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7629cdc4f54748038af3ecc53450517c, State: Initialized, Role: FOLLOWER
I20260619 16:46:34.380098 9092 consensus_queue.cc:260] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.380098 9091 consensus_queue.cc:260] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.381060 9091 ts_tablet_manager.cc:1428] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:46:34.381060 9092 ts_tablet_manager.cc:1428] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 16:46:34.381421 9074 heartbeater.cc:503] Master 127.3.233.190:35777 was elected leader, sending a full tablet report...
I20260619 16:46:34.381583 8812 heartbeater.cc:503] Master 127.3.233.190:35777 was elected leader, sending a full tablet report...
I20260619 16:46:34.381996 9094 raft_consensus.cc:348] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.382229 9094 raft_consensus.cc:374] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:34.382282 9094 raft_consensus.cc:729] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a4cccf0ee7483cb1dda42b52696b64, State: Initialized, Role: FOLLOWER
I20260619 16:46:34.382733 9094 consensus_queue.cc:260] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.383605 9094 ts_tablet_manager.cc:1428] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:46:34.383632 8943 heartbeater.cc:503] Master 127.3.233.190:35777 was elected leader, sending a full tablet report...
I20260619 16:46:34.389921 9098 raft_consensus.cc:482] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:34.390069 9098 raft_consensus.cc:504] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.391073 9098 leader_election.cc:288] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158a55b7e76d4443935a327ff085e33d (127.3.233.131:45695), 25a4cccf0ee7483cb1dda42b52696b64 (127.3.233.130:42961)
I20260619 16:46:34.392928 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.391414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:48211 (local address 127.3.233.131:45695)
0619 16:46:34.391547 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.391552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.391569 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:34.391610 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.391613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.391652 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.391733 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.392505 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.392513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.392519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.392535 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.392559 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.392569 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.392645 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.392758 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.392796 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:46:34.392962 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.391292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:45695 (local address 127.3.233.129:48211)
0619 16:46:34.391436 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.391453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.391527 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.391775 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.391779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.391794 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:34.392342 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:34.392345 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:34.392355 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:34.392385 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:34.392658 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:34.392662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:34.392667 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:46:34.392751 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:46:34.392789 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:46:34.393456 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.391645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:55719 (local address 127.3.233.130:42961)
0619 16:46:34.391758 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.391762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.392078 (+ 316us) server_negotiation.cc:407] Connection header received
0619 16:46:34.392164 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.392166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.392217 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.392326 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.393011 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.393017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.393023 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.393035 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.393053 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.393064 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.393167 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.393298 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.393344 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:46:34.393573 9028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc324de75a9243beba8adf9239e400e7" candidate_uuid: "7629cdc4f54748038af3ecc53450517c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158a55b7e76d4443935a327ff085e33d" is_pre_election: true
I20260619 16:46:34.393765 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.391566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:42961 (local address 127.3.233.129:55719)
0619 16:46:34.391959 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.391995 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.392087 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.392337 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.392340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.392361 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:34.392850 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:46:34.392852 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:34.392870 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:46:34.392894 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:34.393172 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:34.393176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:34.393179 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:46:34.393600 (+ 421us) client_negotiation.cc:240] Negotiation successful
0619 16:46:34.393637 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":107,"threads_started":1}
I20260619 16:46:34.393851 9028 raft_consensus.cc:2454] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7629cdc4f54748038af3ecc53450517c in term 0.
I20260619 16:46:34.394323 8700 leader_election.cc:302] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158a55b7e76d4443935a327ff085e33d, 7629cdc4f54748038af3ecc53450517c; no voters:
I20260619 16:46:34.394367 8897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc324de75a9243beba8adf9239e400e7" candidate_uuid: "7629cdc4f54748038af3ecc53450517c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a4cccf0ee7483cb1dda42b52696b64" is_pre_election: true
I20260619 16:46:34.394559 9098 raft_consensus.cc:2781] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:34.394642 9098 raft_consensus.cc:482] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:34.394616 8897 raft_consensus.cc:2454] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7629cdc4f54748038af3ecc53450517c in term 0.
I20260619 16:46:34.394696 9098 raft_consensus.cc:3037] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:34.395879 9098 raft_consensus.cc:504] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
I20260619 16:46:34.396250 9098 leader_election.cc:288] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [CANDIDATE]: Term 1 election: Requested vote from peers 158a55b7e76d4443935a327ff085e33d (127.3.233.131:45695), 25a4cccf0ee7483cb1dda42b52696b64 (127.3.233.130:42961)
I20260619 16:46:34.396667 8897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc324de75a9243beba8adf9239e400e7" candidate_uuid: "7629cdc4f54748038af3ecc53450517c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a4cccf0ee7483cb1dda42b52696b64"
I20260619 16:46:34.396720 9028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc324de75a9243beba8adf9239e400e7" candidate_uuid: "7629cdc4f54748038af3ecc53450517c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158a55b7e76d4443935a327ff085e33d"
I20260619 16:46:34.396826 8897 raft_consensus.cc:3037] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:34.396834 9028 raft_consensus.cc:3037] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:34.397837 9028 raft_consensus.cc:2454] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7629cdc4f54748038af3ecc53450517c in term 1.
I20260619 16:46:34.397858 8897 raft_consensus.cc:2454] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7629cdc4f54748038af3ecc53450517c in term 1.
I20260619 16:46:34.398125 8700 leader_election.cc:302] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158a55b7e76d4443935a327ff085e33d, 7629cdc4f54748038af3ecc53450517c; no voters:
I20260619 16:46:34.398335 9098 raft_consensus.cc:2781] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:34.398586 9098 raft_consensus.cc:686] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 1 LEADER]: Becoming Leader. State: Replica: 7629cdc4f54748038af3ecc53450517c, State: Running, Role: LEADER
I20260619 16:46:34.398989 9098 consensus_queue.cc:237] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } }
W20260619 16:46:34.399297 8944 tablet.cc:2369] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:34.402320 8624 catalog_manager.cc:5495] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7629cdc4f54748038af3ecc53450517c (127.3.233.129). New cstate: current_term: 1 leader_uuid: "7629cdc4f54748038af3ecc53450517c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7629cdc4f54748038af3ecc53450517c" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41283 } health_report { overall_health: HEALTHY } } }
I20260619 16:46:34.412828 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.411277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.3.233.129:41283)
0619 16:46:34.411444 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.411448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.411477 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:46:34.411550 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.411552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.411610 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.411674 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.412526 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.412528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.412530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.412540 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.412558 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.412566 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.412638 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.412698 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.412730 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:46:34.418877 8897 raft_consensus.cc:1261] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 1 FOLLOWER]: Refusing update from remote peer 7629cdc4f54748038af3ecc53450517c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:34.418891 9028 raft_consensus.cc:1261] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 1 FOLLOWER]: Refusing update from remote peer 7629cdc4f54748038af3ecc53450517c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:34.419672 9101 consensus_queue.cc:1035] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25a4cccf0ee7483cb1dda42b52696b64" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 42961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:34.419893 9098 consensus_queue.cc:1035] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [LEADER]: Connected to new peer: Peer: permanent_uuid: "158a55b7e76d4443935a327ff085e33d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:34.435936 8623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:34.436092 8623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:34.438652 8623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13da73b472a94ad6b2fd55a99de50f79: Sending DeleteTablet for 3 replicas of tablet dc324de75a9243beba8adf9239e400e7
I20260619 16:46:34.439535 8746 tablet_service.cc:1514] Processing DeleteTablet for tablet dc324de75a9243beba8adf9239e400e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:34 UTC) from {username='slave'} at 127.0.0.1:55472
I20260619 16:46:34.439544 8877 tablet_service.cc:1514] Processing DeleteTablet for tablet dc324de75a9243beba8adf9239e400e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:34 UTC) from {username='slave'} at 127.0.0.1:54986
I20260619 16:46:34.440084 9115 tablet_replica.cc:330] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c: stopping tablet replica
I20260619 16:46:34.440071 9008 tablet_service.cc:1514] Processing DeleteTablet for tablet dc324de75a9243beba8adf9239e400e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:34 UTC) from {username='slave'} at 127.0.0.1:34858
I20260619 16:46:34.440470 9116 tablet_replica.cc:330] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: stopping tablet replica
I20260619 16:46:34.440644 9115 raft_consensus.cc:2229] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:34.440750 9116 raft_consensus.cc:2229] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:34.441035 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8684
I20260619 16:46:34.441074 9116 raft_consensus.cc:2258] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:34.441216 9115 raft_consensus.cc:2258] T dc324de75a9243beba8adf9239e400e7 P 7629cdc4f54748038af3ecc53450517c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:34.441576 9114 tablet_replica.cc:330] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: stopping tablet replica
I20260619 16:46:34.441895 9114 raft_consensus.cc:2229] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:34.442214 9114 raft_consensus.cc:2258] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:34.443565 9116 ts_tablet_manager.cc:1905] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:34.443845 9114 ts_tablet_manager.cc:1905] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:34.445802 9116 ts_tablet_manager.cc:1918] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:34.445912 9116 log.cc:1196] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/dc324de75a9243beba8adf9239e400e7
I20260619 16:46:34.446236 9116 ts_tablet_manager.cc:1939] T dc324de75a9243beba8adf9239e400e7 P 158a55b7e76d4443935a327ff085e33d: Deleting consensus metadata
I20260619 16:46:34.447175 9114 ts_tablet_manager.cc:1918] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:34.447269 9114 log.cc:1196] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/dc324de75a9243beba8adf9239e400e7
I20260619 16:46:34.447657 8609 catalog_manager.cc:4841] TS 158a55b7e76d4443935a327ff085e33d (127.3.233.131:45695): tablet dc324de75a9243beba8adf9239e400e7 (table test-table [id=5c374c4c9d504d88b1182ae83bf942b7]) successfully deleted
I20260619 16:46:34.447727 9114 ts_tablet_manager.cc:1939] T dc324de75a9243beba8adf9239e400e7 P 25a4cccf0ee7483cb1dda42b52696b64: Deleting consensus metadata
I20260619 16:46:34.449214 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8815
I20260619 16:46:34.449215 8609 catalog_manager.cc:4841] TS 25a4cccf0ee7483cb1dda42b52696b64 (127.3.233.130:42961): tablet dc324de75a9243beba8adf9239e400e7 (table test-table [id=5c374c4c9d504d88b1182ae83bf942b7]) successfully deleted
W20260619 16:46:34.449288 8610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:41283 (error 108)
I20260619 16:46:34.451583 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.451311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:41283 (local address 127.0.0.1:55486)
0619 16:46:34.451443 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.451505 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:41283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 16:46:34.451831 8610 catalog_manager.cc:4584] TS 7629cdc4f54748038af3ecc53450517c (127.3.233.129:41283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc324de75a9243beba8adf9239e400e7: Network error: Client connection negotiation failed: client connection to 127.3.233.129:41283: connect: Connection refused (error 111)
I20260619 16:46:34.455076 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8946
I20260619 16:46:34.461393 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 8593
2026-06-19T16:46:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (923 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 16:46:34.483656 4006 mini_oidc.cc:152] Starting JWKS server
I20260619 16:46:34.485235 4006 webserver.cc:460] Webserver started at https://127.0.0.1:37227/ using document root <none> and password file <none>
I20260619 16:46:34.485428 4006 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:46:34.485702 4006 webserver.cc:460] Webserver started at http://127.0.0.1:44231/ using document root <none> and password file <none>
2026-06-19T16:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:34Z Disabled control of system clock
I20260619 16:46:34.494601 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:45731
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:35573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:45731
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37227/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/testchainca.cert with env {}
W20260619 16:46:34.605427 9123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:34.605710 9123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:34.605763 9123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:34.610116 9123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:34.610184 9123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:34.610200 9123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:34.610217 9123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:34.610237 9123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:46:34.610278 9123 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:46:34.610307 9123 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37227/jwks.json
I20260619 16:46:34.616041 9123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:45731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:45731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37227/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:34.617167 9123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:34.618230 9123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:34.626773 9128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:34.626899 9131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.626832 9123 server_base.cc:1034] running on GCE node
W20260619 16:46:34.626783 9129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.627579 9123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:34.628695 9123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:34.629889 9123 hybrid_clock.cc:648] HybridClock initialized: now 1781887594629873 us; error 32 us; skew 500 ppm
I20260619 16:46:34.631881 9123 webserver.cc:460] Webserver started at http://127.3.233.190:40969/ using document root <none> and password file <none>
I20260619 16:46:34.632458 9123 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:34.632550 9123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:34.632776 9123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:34.635568 9123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "ea383f509feb4f439070cb965c312b5e"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "ee26b1b352c8b41a4d8e9f5ec011a8ca"
server_key_iv: "6c336c0e91fca22cde71c05d94725d59"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.636157 9123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "ea383f509feb4f439070cb965c312b5e"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "ee26b1b352c8b41a4d8e9f5ec011a8ca"
server_key_iv: "6c336c0e91fca22cde71c05d94725d59"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.639909 9123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:46:34.642307 9137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.643682 9123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:34.643862 9123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "ea383f509feb4f439070cb965c312b5e"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "ee26b1b352c8b41a4d8e9f5ec011a8ca"
server_key_iv: "6c336c0e91fca22cde71c05d94725d59"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.643971 9123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:34.654913 9123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:34.655707 9123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:34.655916 9123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:34.663257 9123 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:45731
I20260619 16:46:34.663268 9189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:45731 every 8 connection(s)
I20260619 16:46:34.664222 9123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:34.667492 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:34.670583 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9123
I20260619 16:46:34.670715 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:34.671047 4006 external_mini_cluster.cc:1428] Setting key c40c9b9978e29e3067a4b574ea3b82e0
I20260619 16:46:34.673720 9190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e: Bootstrap starting.
I20260619 16:46:34.676003 9190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:34.676941 9190 log.cc:824] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:34.679204 9190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e: No bootstrap required, opened a new log
I20260619 16:46:34.680728 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.672516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.3.233.190:45731)
0619 16:46:34.673161 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.673176 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.673211 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:46:34.673920 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.673944 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.674337 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.674674 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.675143 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.676073 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.677294 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.677955 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.678074 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:34.678967 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.678979 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.678991 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.679422 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.679475 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.679495 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.679576 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.679732 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.680047 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":129,"threads_started":1}
I20260619 16:46:34.682827 9190 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea383f509feb4f439070cb965c312b5e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45731 } }
I20260619 16:46:34.683094 9190 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:34.683167 9190 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea383f509feb4f439070cb965c312b5e, State: Initialized, Role: FOLLOWER
I20260619 16:46:34.683494 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:34.683646 9190 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea383f509feb4f439070cb965c312b5e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45731 } }
I20260619 16:46:34.683797 9190 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:34.683858 9190 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:34.683961 9190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:34.684963 9190 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea383f509feb4f439070cb965c312b5e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45731 } }
I20260619 16:46:34.685304 9190 leader_election.cc:302] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea383f509feb4f439070cb965c312b5e; no voters:
I20260619 16:46:34.685645 9190 leader_election.cc:288] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:34.685719 9195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:34.685967 9195 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [term 1 LEADER]: Becoming Leader. State: Replica: ea383f509feb4f439070cb965c312b5e, State: Running, Role: LEADER
I20260619 16:46:34.686260 9195 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea383f509feb4f439070cb965c312b5e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45731 } }
I20260619 16:46:34.687028 9190 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:34.687960 9197 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea383f509feb4f439070cb965c312b5e. Latest consensus state: current_term: 1 leader_uuid: "ea383f509feb4f439070cb965c312b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea383f509feb4f439070cb965c312b5e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45731 } } }
I20260619 16:46:34.688115 9197 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:34.688493 9204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:34.688446 9196 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea383f509feb4f439070cb965c312b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea383f509feb4f439070cb965c312b5e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45731 } } }
I20260619 16:46:34.688580 9196 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:34.691377 9204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:34.696956 9204 catalog_manager.cc:1347] Generated new cluster ID: 50b0388ddec94b688612cef772d5c3a5
I20260619 16:46:34.697046 9204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:34.715391 9204 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:34.716348 9204 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:34.722095 9204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea383f509feb4f439070cb965c312b5e: Generated new TSK 0
I20260619 16:46:34.723017 9204 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:46:34.750742 9220 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 16:46:34.750911 9220 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:46:34.751472 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.734319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.3.233.190:45731)
0619 16:46:34.734459 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.734466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.734683 (+ 217us) server_negotiation.cc:407] Connection header received
0619 16:46:34.734967 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.734973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.735028 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.735117 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:46:34.735407 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.735981 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.737216 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.737441 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.737520 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:34.737911 (+ 391us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:46:34.751097 (+ 13186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37227/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:46:34.751355 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37227/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:46:34.751574 9193 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37227/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 16:46:34.752992 9219 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:34.734222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:45731 (local address 127.0.0.1:34086)
0619 16:46:34.734591 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:34.734607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:34.734745 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:34.735165 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:34.735169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:34.735177 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:46:34.735304 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:34.735310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.736083 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.736089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:34.737094 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:34.737105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.737556 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.737560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:34.737659 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:34.737719 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:46:34.751353 (+ 13634us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37227/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:46:34.751648 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.233.190:45731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37227/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.233.190:45731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37227/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 16:46:34.753870 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9123
2026-06-19T16:46:34Z chronyd exiting
I20260619 16:46:34.791270 4006 test_util.cc:175] -----------------------------------------------
I20260619 16:46:34.791386 4006 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887569513258-4006-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (309 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 16:46:34.792378 4006 mini_oidc.cc:152] Starting JWKS server
I20260619 16:46:34.793205 4006 webserver.cc:460] Webserver started at https://127.0.0.1:43937/ using document root <none> and password file <none>
I20260619 16:46:34.793280 4006 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:46:34.793542 4006 webserver.cc:460] Webserver started at http://127.0.0.1:33195/ using document root <none> and password file <none>
2026-06-19T16:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:34Z Disabled control of system clock
I20260619 16:46:34.801878 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34925
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34925
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43937/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/testchainca.cert with env {}
W20260619 16:46:34.913091 9226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:34.913404 9226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:34.913480 9226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:34.917773 9226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:34.917853 9226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:34.917882 9226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:34.917923 9226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:34.917970 9226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:46:34.918011 9226 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:46:34.918041 9226 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43937/jwks.json
I20260619 16:46:34.923650 9226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43937/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:34.924731 9226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:34.925745 9226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:34.934058 9231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:34.934090 9232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.934194 9226 server_base.cc:1034] running on GCE node
W20260619 16:46:34.934404 9234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:34.934916 9226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:34.936093 9226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:34.937288 9226 hybrid_clock.cc:648] HybridClock initialized: now 1781887594937256 us; error 45 us; skew 500 ppm
I20260619 16:46:34.939254 9226 webserver.cc:460] Webserver started at http://127.3.233.190:43045/ using document root <none> and password file <none>
I20260619 16:46:34.939851 9226 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:34.939944 9226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:34.940163 9226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:34.942163 9226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "627e825a16667100f9e9e2ea4f37d024"
server_key_iv: "a63e9c3da6d0b8e69d2145ce668b8303"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.942760 9226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "627e825a16667100f9e9e2ea4f37d024"
server_key_iv: "a63e9c3da6d0b8e69d2145ce668b8303"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.946429 9226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:34.949245 9240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:34.950652 9226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:34.950882 9226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4"
format_stamp: "Formatted at 2026-06-19 16:46:34 on dist-test-slave-cgwx"
server_key: "627e825a16667100f9e9e2ea4f37d024"
server_key_iv: "a63e9c3da6d0b8e69d2145ce668b8303"
server_key_version: "encryptionkey@0"
I20260619 16:46:34.951089 9226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:34.967003 9226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:34.967792 9226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:34.968008 9226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:34.975235 9226 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34925
I20260619 16:46:34.975243 9292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34925 every 8 connection(s)
I20260619 16:46:34.976225 9226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:34.977581 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9226
I20260619 16:46:34.977711 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:34.978016 4006 external_mini_cluster.cc:1428] Setting key 4854a8703c4c5b2ad3c3c8c0651dfa0e
I20260619 16:46:34.980073 9293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:34.986944 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4: Bootstrap starting.
I20260619 16:46:34.987977 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:34.979620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.3.233.190:34925)
0619 16:46:34.980308 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:34.980324 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:34.980360 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:46:34.981008 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:34.981025 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:34.981423 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:34.981710 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:34.982062 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.983149 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.984334 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:34.984816 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:34.984946 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:34.985923 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:34.985929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:34.985943 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:34.986361 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:34.986449 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:34.986471 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:34.986553 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:34.986832 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 16:46:34.987210 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
I20260619 16:46:34.989599 9293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:34.990527 9293 log.cc:824] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:34.991175 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:34.992900 9293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4: No bootstrap required, opened a new log
I20260619 16:46:34.996680 9293 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34925 } }
I20260619 16:46:34.996922 9293 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:34.996984 9293 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0508ebf9d8d4d46a07bc7aee1821ef4, State: Initialized, Role: FOLLOWER
I20260619 16:46:34.997471 9293 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34925 } }
I20260619 16:46:34.997596 9293 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:34.997673 9293 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:34.997774 9293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:34.998708 9293 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34925 } }
I20260619 16:46:34.999094 9293 leader_election.cc:302] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0508ebf9d8d4d46a07bc7aee1821ef4; no voters:
I20260619 16:46:34.999439 9293 leader_election.cc:288] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:34.999565 9298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:34.999802 9298 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [term 1 LEADER]: Becoming Leader. State: Replica: a0508ebf9d8d4d46a07bc7aee1821ef4, State: Running, Role: LEADER
I20260619 16:46:35.000123 9298 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34925 } }
I20260619 16:46:35.000676 9293 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:35.001886 9300 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0508ebf9d8d4d46a07bc7aee1821ef4. Latest consensus state: current_term: 1 leader_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34925 } } }
I20260619 16:46:35.002024 9300 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:35.001886 9299 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0508ebf9d8d4d46a07bc7aee1821ef4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34925 } } }
I20260619 16:46:35.002354 9299 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:35.002514 9307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:35.003190 9314 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:46:35.006027 9307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:35.011847 9307 catalog_manager.cc:1347] Generated new cluster ID: 975fcc33bdde4a979a1bd670f827f414
I20260619 16:46:35.011915 9307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:35.019767 9307 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:35.020982 9307 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:35.031181 9307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0508ebf9d8d4d46a07bc7aee1821ef4: Generated new TSK 0
I20260619 16:46:35.032044 9307 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:46:35.064141 9323 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 16:46:35.064251 9323 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:46:35.064927 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.041838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.3.233.190:34925)
0619 16:46:35.041984 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:35.041987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:35.042111 (+ 124us) server_negotiation.cc:407] Connection header received
0619 16:46:35.042273 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:35.042276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:35.042324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:35.042420 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:46:35.042670 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.043271 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.044530 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.044730 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.044805 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.045127 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:46:35.064570 (+ 19443us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:46:35.064812 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 16:46:35.065012 9296 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 16:46:35.065197 9322 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:35.041677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34925 (local address 127.0.0.1:58252)
0619 16:46:35.042013 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:35.042028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:35.042146 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:35.042444 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:35.042447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:35.042455 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:46:35.042566 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:35.042573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.043398 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.043402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:35.044407 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:35.044417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.044840 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.044843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:35.044937 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.044965 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:46:35.064794 (+ 19829us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:46:35.065014 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.233.190:34925: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43937/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260619 16:46:35.065826 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9226
2026-06-19T16:46:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 16:46:35.105166 4006 mini_oidc.cc:152] Starting JWKS server
I20260619 16:46:35.106030 4006 webserver.cc:460] Webserver started at https://127.0.0.1:42191/ using document root <none> and password file <none>
I20260619 16:46:35.106158 4006 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:46:35.106451 4006 webserver.cc:460] Webserver started at http://127.0.0.1:42087/ using document root <none> and password file <none>
2026-06-19T16:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:35Z Disabled control of system clock
I20260619 16:46:35.115203 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38011
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:38011
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42191/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:46:35.225733 9329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:35.226012 9329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:35.226051 9329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:35.230180 9329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:35.230243 9329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:35.230259 9329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:35.230278 9329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:35.230293 9329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:46:35.230360 9329 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:46:35.230384 9329 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42191/jwks.json
I20260619 16:46:35.235206 9329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:38011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42191/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:35.236313 9329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:35.237695 9329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:35.246306 9334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:35.246292 9335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:35.246291 9337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:35.246688 9329 server_base.cc:1034] running on GCE node
I20260619 16:46:35.247341 9329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:35.248477 9329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:35.249670 9329 hybrid_clock.cc:648] HybridClock initialized: now 1781887595249650 us; error 34 us; skew 500 ppm
I20260619 16:46:35.251677 9329 webserver.cc:460] Webserver started at http://127.3.233.190:46049/ using document root <none> and password file <none>
I20260619 16:46:35.252254 9329 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:35.252357 9329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:35.252590 9329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:35.254626 9329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "6a591419637942de9b82603b12a40d72"
format_stamp: "Formatted at 2026-06-19 16:46:35 on dist-test-slave-cgwx"
server_key: "440d9b1d9c86f30b448318c4a6649b4f"
server_key_iv: "3eb5e2da49d9ccfeea41fba40215ad9f"
server_key_version: "encryptionkey@0"
I20260619 16:46:35.255192 9329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "6a591419637942de9b82603b12a40d72"
format_stamp: "Formatted at 2026-06-19 16:46:35 on dist-test-slave-cgwx"
server_key: "440d9b1d9c86f30b448318c4a6649b4f"
server_key_iv: "3eb5e2da49d9ccfeea41fba40215ad9f"
server_key_version: "encryptionkey@0"
I20260619 16:46:35.258805 9329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:46:35.261292 9343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:35.262502 9329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:35.262645 9329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "6a591419637942de9b82603b12a40d72"
format_stamp: "Formatted at 2026-06-19 16:46:35 on dist-test-slave-cgwx"
server_key: "440d9b1d9c86f30b448318c4a6649b4f"
server_key_iv: "3eb5e2da49d9ccfeea41fba40215ad9f"
server_key_version: "encryptionkey@0"
I20260619 16:46:35.262773 9329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:35.285928 9329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:35.286669 9329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:35.286871 9329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:35.294041 9329 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:38011
I20260619 16:46:35.294078 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:38011 every 8 connection(s)
I20260619 16:46:35.294997 9329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:35.298048 9396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:35.301151 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9329
I20260619 16:46:35.301255 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:35.301528 4006 external_mini_cluster.cc:1428] Setting key 6e27b137b6acd9216ea932ee8c4eb165
I20260619 16:46:35.303861 9396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72: Bootstrap starting.
I20260619 16:46:35.306126 9396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:35.307010 9396 log.cc:824] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:35.309075 9396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72: No bootstrap required, opened a new log
I20260619 16:46:35.311123 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.303105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39248 (local address 127.3.233.190:38011)
0619 16:46:35.303753 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:35.303785 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:35.303820 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:46:35.304486 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:35.304503 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:35.304954 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:35.305239 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:35.305577 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.306636 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.307802 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.308331 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.308450 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.309323 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:35.309328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:35.309340 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:35.309715 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:35.309769 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:35.309788 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:35.309863 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:35.310014 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 16:46:35.310333 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":158,"threads_started":1}
I20260619 16:46:35.312790 9396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a591419637942de9b82603b12a40d72" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38011 } }
I20260619 16:46:35.313027 9396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:35.313107 9396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a591419637942de9b82603b12a40d72, State: Initialized, Role: FOLLOWER
I20260619 16:46:35.313525 9396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a591419637942de9b82603b12a40d72" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38011 } }
I20260619 16:46:35.313642 9396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:35.313758 9396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:35.313864 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:35.313874 9396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:35.315014 9396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a591419637942de9b82603b12a40d72" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38011 } }
I20260619 16:46:35.315343 9396 leader_election.cc:302] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a591419637942de9b82603b12a40d72; no voters:
I20260619 16:46:35.315675 9396 leader_election.cc:288] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:35.315800 9401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:35.315990 9401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [term 1 LEADER]: Becoming Leader. State: Replica: 6a591419637942de9b82603b12a40d72, State: Running, Role: LEADER
I20260619 16:46:35.316269 9401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a591419637942de9b82603b12a40d72" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38011 } }
I20260619 16:46:35.316882 9396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:35.317970 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a591419637942de9b82603b12a40d72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a591419637942de9b82603b12a40d72" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38011 } } }
I20260619 16:46:35.318009 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a591419637942de9b82603b12a40d72. Latest consensus state: current_term: 1 leader_uuid: "6a591419637942de9b82603b12a40d72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a591419637942de9b82603b12a40d72" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38011 } } }
I20260619 16:46:35.318112 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:35.318111 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:35.318516 9410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:35.321637 9410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:35.326596 9410 catalog_manager.cc:1347] Generated new cluster ID: ad05b5b134d24d2f8da7d25a222410de
I20260619 16:46:35.326659 9410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:35.351431 9410 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:35.352396 9410 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:35.360067 9410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a591419637942de9b82603b12a40d72: Generated new TSK 0
I20260619 16:46:35.360775 9410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:35.378957 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.373750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.3.233.190:38011)
0619 16:46:35.373904 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:35.373909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:35.374053 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:46:35.374280 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:35.374283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:35.374334 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:35.374460 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:46:35.374732 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.375405 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.376365 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.376623 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.376742 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.377072 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:46:35.378538 (+ 1466us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:46:35.378773 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 16:46:35.379029 9399 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39256: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 16:46:35.380971 9425 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:35.373633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38011 (local address 127.0.0.1:39256)
0619 16:46:35.373946 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:35.373966 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:35.374109 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:35.374465 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:35.374469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:35.374477 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:46:35.374611 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:35.374617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.375527 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.375533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:35.376238 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:35.376246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.376759 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.376764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:35.376858 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.376880 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:46:35.378713 (+ 1833us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:46:35.380793 (+ 2080us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.233.190:38011: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":1887,"thread_start_us":99,"threads_started":1}
I20260619 16:46:35.381536 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9329
E20260619 16:46:35.392812 9426 webserver.cc:533] Webserver: sslize failed:
E20260619 16:46:35.392910 9426 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T16:46:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T16:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:35Z Disabled control of system clock
I20260619 16:46:35.430428 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38249
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:38249
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/testchainca.cert with env {}
W20260619 16:46:35.539788 9430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:35.540076 9430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:35.540117 9430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:35.544133 9430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:35.544196 9430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:35.544211 9430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:35.544229 9430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:35.544245 9430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:46:35.544260 9430 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:46:35.544274 9430 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 16:46:35.549634 9430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:38249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:35.550673 9430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:35.551730 9430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:35.560091 9435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:35.560102 9438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:35.560102 9436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:35.560278 9430 server_base.cc:1034] running on GCE node
I20260619 16:46:35.560873 9430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:35.561951 9430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:35.563136 9430 hybrid_clock.cc:648] HybridClock initialized: now 1781887595563109 us; error 49 us; skew 500 ppm
I20260619 16:46:35.564954 9430 webserver.cc:460] Webserver started at http://127.3.233.190:35979/ using document root <none> and password file <none>
I20260619 16:46:35.565562 9430 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:35.565652 9430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:35.565881 9430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:35.567955 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "d043a783277849a6ae1a5b1475866769"
format_stamp: "Formatted at 2026-06-19 16:46:35 on dist-test-slave-cgwx"
server_key: "48493fb609067b85893cab103babf41c"
server_key_iv: "14ccf802a655ed39eab7581848885497"
server_key_version: "encryptionkey@0"
I20260619 16:46:35.568509 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "d043a783277849a6ae1a5b1475866769"
format_stamp: "Formatted at 2026-06-19 16:46:35 on dist-test-slave-cgwx"
server_key: "48493fb609067b85893cab103babf41c"
server_key_iv: "14ccf802a655ed39eab7581848885497"
server_key_version: "encryptionkey@0"
I20260619 16:46:35.572036 9430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:46:35.574339 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:35.575613 9430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:46:35.575748 9430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "d043a783277849a6ae1a5b1475866769"
format_stamp: "Formatted at 2026-06-19 16:46:35 on dist-test-slave-cgwx"
server_key: "48493fb609067b85893cab103babf41c"
server_key_iv: "14ccf802a655ed39eab7581848885497"
server_key_version: "encryptionkey@0"
I20260619 16:46:35.575860 9430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:35.591751 9430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:35.592547 9430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:35.592780 9430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:35.601060 9430 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:38249
I20260619 16:46:35.601089 9496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:38249 every 8 connection(s)
I20260619 16:46:35.602599 9430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:35.606106 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9430
I20260619 16:46:35.606272 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:35.606588 4006 external_mini_cluster.cc:1428] Setting key 6263159c232c51afa316813a1181de36
I20260619 16:46:35.607558 9497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:35.614805 9497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769: Bootstrap starting.
I20260619 16:46:35.617314 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.608304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.3.233.190:38249)
0619 16:46:35.609137 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:35.609163 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:35.609216 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:46:35.610181 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:35.610201 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:35.610622 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:35.611417 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:35.611769 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.612775 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.613744 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.614254 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.614439 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.615418 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:35.615424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:35.615437 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:46:35.615830 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:35.615941 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:46:35.615964 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:35.616076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:35.616208 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 16:46:35.616539 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":157,"threads_started":1}
I20260619 16:46:35.617460 9497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:35.618273 9497 log.cc:824] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:35.620448 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:46:35.620743 9497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769: No bootstrap required, opened a new log
I20260619 16:46:35.625023 9497 raft_consensus.cc:348] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d043a783277849a6ae1a5b1475866769" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38249 } }
I20260619 16:46:35.625289 9497 raft_consensus.cc:374] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:35.625391 9497 raft_consensus.cc:729] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d043a783277849a6ae1a5b1475866769, State: Initialized, Role: FOLLOWER
I20260619 16:46:35.625866 9497 consensus_queue.cc:260] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d043a783277849a6ae1a5b1475866769" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38249 } }
I20260619 16:46:35.625999 9497 raft_consensus.cc:388] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:35.626082 9497 raft_consensus.cc:482] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:35.626209 9497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:35.627465 9497 raft_consensus.cc:504] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d043a783277849a6ae1a5b1475866769" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38249 } }
I20260619 16:46:35.627859 9497 leader_election.cc:302] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d043a783277849a6ae1a5b1475866769; no voters:
I20260619 16:46:35.628250 9497 leader_election.cc:288] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:35.628382 9502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:35.628629 9502 raft_consensus.cc:686] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [term 1 LEADER]: Becoming Leader. State: Replica: d043a783277849a6ae1a5b1475866769, State: Running, Role: LEADER
I20260619 16:46:35.628947 9502 consensus_queue.cc:237] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d043a783277849a6ae1a5b1475866769" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38249 } }
I20260619 16:46:35.629645 9497 sys_catalog.cc:564] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:35.630815 9503 sys_catalog.cc:455] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d043a783277849a6ae1a5b1475866769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d043a783277849a6ae1a5b1475866769" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38249 } } }
I20260619 16:46:35.631017 9503 sys_catalog.cc:458] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:35.631299 9504 sys_catalog.cc:455] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d043a783277849a6ae1a5b1475866769. Latest consensus state: current_term: 1 leader_uuid: "d043a783277849a6ae1a5b1475866769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d043a783277849a6ae1a5b1475866769" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38249 } } }
I20260619 16:46:35.631421 9504 sys_catalog.cc:458] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:35.631546 9511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:35.634420 9511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:35.640762 9511 catalog_manager.cc:1347] Generated new cluster ID: a6f6310621094b1dab8c0ef5413253f4
I20260619 16:46:35.640864 9511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:35.656741 9511 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:35.658175 9511 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:35.680365 9511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d043a783277849a6ae1a5b1475866769: Generated new TSK 0
I20260619 16:46:35.681259 9511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:35.758255 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:35.695110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.3.233.190:38249)
0619 16:46:35.695248 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:35.695252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:35.695450 (+ 198us) server_negotiation.cc:407] Connection header received
0619 16:46:35.695650 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:35.695653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:35.695708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:35.695822 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:46:35.696065 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.696606 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.697476 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.697704 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.697810 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.698141 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:46:35.757713 (+ 59572us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:46:35.758102 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 16:46:35.758340 9500 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 16:46:35.758373 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:35.695039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38249 (local address 127.0.0.1:36204)
0619 16:46:35.695334 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:35.695347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:35.695502 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:35.695819 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:35.695823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:35.695832 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:46:35.695945 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:35.695952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.696718 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.696723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:35.697342 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:35.697349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:35.697834 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:35.697838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:35.697937 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:35.697960 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:46:35.757998 (+ 60038us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:46:35.758189 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.233.190:38249: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":119,"threads_started":1}
I20260619 16:46:35.758852 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9430
2026-06-19T16:46:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (355 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:36 dist-test-slave-cgwx krb5kdc[9538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:36 dist-test-slave-cgwx krb5kdc[9538](info): set up 2 sockets
Jun 19 16:46:36 dist-test-slave-cgwx krb5kdc[9538](info): commencing operation
krb5kdc: starting...
W20260619 16:46:37.119639 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:37 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:37.238422 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:41945
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:41945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:37.349961 9551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:37.350252 9551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:37.350293 9551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:37.354332 9551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:37.354408 9551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:37.354429 9551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:37.354446 9551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:37.354461 9551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:37.359787 9551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:41945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:37.360833 9551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:37.361805 9551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:37.370513 9556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:37.370517 9557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:37.370545 9559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:37.370806 9551 server_base.cc:1034] running on GCE node
I20260619 16:46:37.371568 9551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:37.372848 9551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:37.374053 9551 hybrid_clock.cc:648] HybridClock initialized: now 1781887597374039 us; error 27 us; skew 500 ppm
Jun 19 16:46:37 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:37.377157 9551 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:37.378119 9551 webserver.cc:460] Webserver started at http://127.3.233.190:36849/ using document root <none> and password file <none>
I20260619 16:46:37.378720 9551 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:37.378818 9551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:37.379108 9551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:37.381327 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "7b4aef6cbc2c4a3092bc8689b75050aa"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "763cc10fb7328d8e3e85a704d3240afc"
server_key_iv: "af33bd422ba8416408eb0cbf92b6eac1"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.381959 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "7b4aef6cbc2c4a3092bc8689b75050aa"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "763cc10fb7328d8e3e85a704d3240afc"
server_key_iv: "af33bd422ba8416408eb0cbf92b6eac1"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.386071 9551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:46:37.388679 9566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:37.390231 9551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:37.390399 9551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "7b4aef6cbc2c4a3092bc8689b75050aa"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "763cc10fb7328d8e3e85a704d3240afc"
server_key_iv: "af33bd422ba8416408eb0cbf92b6eac1"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.390516 9551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:37.413575 9551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:37.416834 9551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:37.417047 9551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:37.424101 9551 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:41945
I20260619 16:46:37.424127 9618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:41945 every 8 connection(s)
I20260619 16:46:37.425060 9551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:37.427765 9619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:37.433068 9619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa: Bootstrap starting.
I20260619 16:46:37.435019 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9551
I20260619 16:46:37.435153 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:37.435446 4006 external_mini_cluster.cc:1428] Setting key 5c16eb259d18a7a414af8d2ef90e20d6
I20260619 16:46:37.435813 9619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:37.436659 9619 log.cc:824] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:37.438799 9619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa: No bootstrap required, opened a new log
I20260619 16:46:37.442215 9619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41945 } }
I20260619 16:46:37.442483 9619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:37.442576 9619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4aef6cbc2c4a3092bc8689b75050aa, State: Initialized, Role: FOLLOWER
I20260619 16:46:37.443126 9619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41945 } }
I20260619 16:46:37.443260 9619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:37.443352 9619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:37.443460 9619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:37.444391 9619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41945 } }
Jun 19 16:46:37 dist-test-slave-cgwx krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:37.444762 9619 leader_election.cc:302] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4aef6cbc2c4a3092bc8689b75050aa; no voters:
I20260619 16:46:37.445159 9619 leader_election.cc:288] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:37.445288 9624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:37.445571 9624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [term 1 LEADER]: Becoming Leader. State: Replica: 7b4aef6cbc2c4a3092bc8689b75050aa, State: Running, Role: LEADER
I20260619 16:46:37.445907 9624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41945 } }
I20260619 16:46:37.446641 9619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:37.447606 9625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41945 } } }
I20260619 16:46:37.447721 9625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:37.448186 9633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:37.449117 9626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b4aef6cbc2c4a3092bc8689b75050aa. Latest consensus state: current_term: 1 leader_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4aef6cbc2c4a3092bc8689b75050aa" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41945 } } }
I20260619 16:46:37.449234 9626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:37.450837 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.436734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.3.233.190:41945)
0619 16:46:37.437285 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:37.437305 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:37.437349 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:46:37.438155 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:37.438175 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:37.438520 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:37.438878 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:37.439706 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.440708 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.441579 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.442070 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.442248 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:37.445079 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:37.445087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:37.445107 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:37.445142 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:37.447397 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:37.448005 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:37.448012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:37.448018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:37.448152 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:37.448490 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:37.448494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:37.448496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:37.449092 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:37.449298 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:37.449566 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:46:37.449893 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":190,"threads_started":1}
I20260619 16:46:37.452366 9633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:37.457965 9633 catalog_manager.cc:1347] Generated new cluster ID: adacfc35bd4448438c5df175fd17bf92
I20260619 16:46:37.458039 9633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:37.489566 9633 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:37.490571 9633 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:37.504842 9633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b4aef6cbc2c4a3092bc8689b75050aa: Generated new TSK 0
I20260619 16:46:37.505652 9633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:37.534109 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:41945
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:37.649880 9645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:37.650192 9645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:37.650274 9645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:37.654624 9645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:37.654726 9645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:37.654830 9645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:37.660449 9645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:41945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:37.661562 9645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:37.662621 9645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:37.672847 9650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:37.672842 9651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:37.673050 9645 server_base.cc:1034] running on GCE node
W20260619 16:46:37.673327 9653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:37.673772 9645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:37.674474 9645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:37.675638 9645 hybrid_clock.cc:648] HybridClock initialized: now 1781887597675621 us; error 37 us; skew 500 ppm
Jun 19 16:46:37 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:37.682466 9645 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:37.683626 9645 webserver.cc:460] Webserver started at http://127.3.233.129:43609/ using document root <none> and password file <none>
I20260619 16:46:37.684222 9645 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:37.684311 9645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:37.684520 9645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:37.686655 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "aef642f663774f05bb5f8bbe643932fc"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "4f824cc01f575cddcb091d27b65f43ca"
server_key_iv: "10735cdca2a3950c6754a1632d763b91"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.687233 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "aef642f663774f05bb5f8bbe643932fc"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "4f824cc01f575cddcb091d27b65f43ca"
server_key_iv: "10735cdca2a3950c6754a1632d763b91"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.691072 9645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:46:37.693822 9660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:37.695493 9645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:37.695598 9645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "aef642f663774f05bb5f8bbe643932fc"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "4f824cc01f575cddcb091d27b65f43ca"
server_key_iv: "10735cdca2a3950c6754a1632d763b91"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.695729 9645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:37.707772 9645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:37.711256 9645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:37.711478 9645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:37.712097 9645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:37.712965 9645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:37.713013 9645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:37.713054 9645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:37.713106 9645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:37.722517 9645 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:40809
I20260619 16:46:37.722558 9773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:40809 every 8 connection(s)
I20260619 16:46:37.723606 9645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:37.731211 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9645
I20260619 16:46:37.731359 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:37.731647 4006 external_mini_cluster.cc:1428] Setting key 65a866ea357d76f7e123370d9c7569e0
Jun 19 16:46:37 dist-test-slave-cgwx krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:37.739481 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.725772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:47701 (local address 127.3.233.190:41945)
0619 16:46:37.725965 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:37.725969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:37.726821 (+ 852us) server_negotiation.cc:407] Connection header received
0619 16:46:37.728392 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:37.728396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:37.728509 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:37.728619 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:37.729841 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.730445 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.731904 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.732221 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.732339 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:37.735262 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:37.735269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:37.735275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:37.735316 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:37.736958 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:37.737654 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:37.737657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:37.737658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:37.737740 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:37.738155 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:37.738158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:37.738159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:37.738348 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:37.738436 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:37.739199 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 16:46:37.739316 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 16:46:37.740731 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.726178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:41945 (local address 127.3.233.129:47701)
0619 16:46:37.726655 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:37.726700 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:37.728153 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:37.728806 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:37.728813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:37.729192 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:37.729684 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:37.729697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.730562 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.730565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:37.731781 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:37.731791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.732377 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.732382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:37.732562 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:37.733225 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:37.733248 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:37.735065 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:37.737083 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:37.737092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:37.737112 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:37.737548 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:37.737839 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:37.737844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:37.737847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:37.738051 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:37.738479 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:37.738488 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:37.738863 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 16:46:37.739143 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 16:46:37.739563 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:37.742107 9774 heartbeater.cc:347] Connected to a master server at 127.3.233.190:41945
I20260619 16:46:37.742403 9774 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:37.743052 9774 heartbeater.cc:511] Master 127.3.233.190:41945 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:37.744741 9583 ts_manager.cc:194] Registered new tserver with Master: aef642f663774f05bb5f8bbe643932fc (127.3.233.129:40809)
I20260619 16:46:37.746587 9583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:47701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:37.761858 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:41945
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:37.874625 9779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:37.874915 9779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:37.875033 9779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:37.879196 9779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:37.879258 9779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:37.879354 9779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:37.884724 9779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:41945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:37.885871 9779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:37.886958 9779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:37.896718 9785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:37.896718 9784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:37.896804 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:37.897176 9779 server_base.cc:1034] running on GCE node
I20260619 16:46:37.897650 9779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:37.898407 9779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:37.899612 9779 hybrid_clock.cc:648] HybridClock initialized: now 1781887597899596 us; error 31 us; skew 500 ppm
Jun 19 16:46:37 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:37.902577 9779 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:37.903663 9779 webserver.cc:460] Webserver started at http://127.3.233.130:46643/ using document root <none> and password file <none>
I20260619 16:46:37.904294 9779 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:37.904382 9779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:37.904614 9779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:37.906711 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "4d11c585388c4d7095db6e1712bced6b"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "9b6a72da0787fb6d7938d7653394004c"
server_key_iv: "e7b8c0e328477450991645d7cf486141"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.907359 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "4d11c585388c4d7095db6e1712bced6b"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "9b6a72da0787fb6d7938d7653394004c"
server_key_iv: "e7b8c0e328477450991645d7cf486141"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.911437 9779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:46:37.914193 9794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:37.915601 9779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:37.915742 9779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "4d11c585388c4d7095db6e1712bced6b"
format_stamp: "Formatted at 2026-06-19 16:46:37 on dist-test-slave-cgwx"
server_key: "9b6a72da0787fb6d7938d7653394004c"
server_key_iv: "e7b8c0e328477450991645d7cf486141"
server_key_version: "encryptionkey@0"
I20260619 16:46:37.915870 9779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:37.940824 9779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:37.944098 9779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:37.944319 9779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:37.944940 9779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:37.945806 9779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:37.945888 9779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:37.945976 9779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:37.946017 9779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:37.956040 9779 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:46495
I20260619 16:46:37.956060 9907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:46495 every 8 connection(s)
I20260619 16:46:37.956970 9779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:37.958346 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9779
I20260619 16:46:37.958487 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:37.958878 4006 external_mini_cluster.cc:1428] Setting key b14058f02dadd1475312fd4f19be2a66
Jun 19 16:46:37 dist-test-slave-cgwx krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:46:37.973223 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.958872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:49533 (local address 127.3.233.190:41945)
0619 16:46:37.959068 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:37.959074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:37.960089 (+ 1015us) server_negotiation.cc:407] Connection header received
0619 16:46:37.961581 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:37.961588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:37.961655 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:37.961764 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:37.963468 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.964003 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.965935 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.966146 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.966231 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:37.969225 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:37.969229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:37.969231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:37.969255 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:37.970631 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:37.971320 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:37.971324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:37.971325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:37.971429 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:37.971802 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:37.971806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:37.971808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:37.972122 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:37.972247 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:37.972923 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 16:46:37.973050 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 16:46:37.974046 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:37.959413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:41945 (local address 127.3.233.130:49533)
0619 16:46:37.959942 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:37.959995 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:37.961349 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:37.961988 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:37.961999 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:37.962527 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:37.963309 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:37.963332 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.964143 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.964148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:37.965771 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:37.965786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:37.966309 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:37.966314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:37.966545 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:37.967361 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:37.967377 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:37.969044 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:37.970752 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:37.970758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:37.970770 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:37.971129 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:37.971547 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:37.971550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:37.971551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:37.971692 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:37.972257 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:37.972263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:37.972577 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 16:46:37.972802 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 16:46:37.973161 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":157,"threads_started":1}
I20260619 16:46:37.975327 9908 heartbeater.cc:347] Connected to a master server at 127.3.233.190:41945
I20260619 16:46:37.975605 9908 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:37.976251 9908 heartbeater.cc:511] Master 127.3.233.190:41945 requested a full tablet report, sending...
I20260619 16:46:37.977499 9583 ts_manager.cc:194] Registered new tserver with Master: 4d11c585388c4d7095db6e1712bced6b (127.3.233.130:46495)
I20260619 16:46:37.978140 9583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:49533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:37.988336 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:41945
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:38.099113 9913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:38.099406 9913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:38.099480 9913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:38.103595 9913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:38.103691 9913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:38.103786 9913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:38.109100 9913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:41945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:38.110204 9913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:38.111260 9913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:38.121023 9921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:38.120988 9918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:38.121001 9919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:38.121459 9913 server_base.cc:1034] running on GCE node
I20260619 16:46:38.121867 9913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:38.122635 9913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:38.123819 9913 hybrid_clock.cc:648] HybridClock initialized: now 1781887598123756 us; error 70 us; skew 500 ppm
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.126652 9913 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:38.127604 9913 webserver.cc:460] Webserver started at http://127.3.233.131:45509/ using document root <none> and password file <none>
I20260619 16:46:38.128225 9913 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:38.128315 9913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:38.128542 9913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:38.130569 9913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "a895265ddbcb4974b8f8ab1497ab7667"
format_stamp: "Formatted at 2026-06-19 16:46:38 on dist-test-slave-cgwx"
server_key: "db3dbe960fd203f0109afc57bc4221aa"
server_key_iv: "6f9c37557ded893794b34a9da499ef0c"
server_key_version: "encryptionkey@0"
I20260619 16:46:38.131186 9913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "a895265ddbcb4974b8f8ab1497ab7667"
format_stamp: "Formatted at 2026-06-19 16:46:38 on dist-test-slave-cgwx"
server_key: "db3dbe960fd203f0109afc57bc4221aa"
server_key_iv: "6f9c37557ded893794b34a9da499ef0c"
server_key_version: "encryptionkey@0"
I20260619 16:46:38.135193 9913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:46:38.138021 9928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:38.139483 9913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:38.139626 9913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "a895265ddbcb4974b8f8ab1497ab7667"
format_stamp: "Formatted at 2026-06-19 16:46:38 on dist-test-slave-cgwx"
server_key: "db3dbe960fd203f0109afc57bc4221aa"
server_key_iv: "6f9c37557ded893794b34a9da499ef0c"
server_key_version: "encryptionkey@0"
I20260619 16:46:38.139747 9913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:38.153050 9913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:38.156531 9913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:38.156742 9913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:38.157382 9913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:38.158226 9913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:38.158322 9913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:38.158396 9913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:38.158456 9913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:38.168074 9913 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:33021
I20260619 16:46:38.168099 10041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:33021 every 8 connection(s)
I20260619 16:46:38.169008 9913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:38.174857 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 9913
I20260619 16:46:38.175012 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:38.175288 4006 external_mini_cluster.cc:1428] Setting key f11794bc25f829da3ab0d67d96680b80
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:38.184031 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.170807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:57643 (local address 127.3.233.190:41945)
0619 16:46:38.171019 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:38.171024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:38.171881 (+ 857us) server_negotiation.cc:407] Connection header received
0619 16:46:38.173409 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:38.173412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:38.173462 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:38.173541 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:38.174854 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.175648 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.177094 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.177347 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.177449 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:38.180108 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:38.180114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:38.180119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:38.180152 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:38.181912 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.182452 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.182456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.182458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.182539 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.183010 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.183014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.183016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.183228 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:38.183315 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:38.183803 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 16:46:38.183912 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:46:38.184924 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.171230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:41945 (local address 127.3.233.131:57643)
0619 16:46:38.171719 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:38.171766 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:38.173189 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:38.173720 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:38.173727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:38.174178 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:38.174690 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:38.174704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.175867 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.175870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:38.176967 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:38.176977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.177527 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.177532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:38.177707 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:38.178395 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:38.178411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:38.179884 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:38.182036 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:38.182041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:38.182055 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:38.182350 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:38.182643 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:38.182646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:38.182648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:38.182805 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:38.183326 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:38.183331 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:38.183590 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 16:46:38.183751 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 16:46:38.184014 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":108,"threads_started":1}
I20260619 16:46:38.186378 10042 heartbeater.cc:347] Connected to a master server at 127.3.233.190:41945
I20260619 16:46:38.186627 10042 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:38.187217 10042 heartbeater.cc:511] Master 127.3.233.190:41945 requested a full tablet report, sending...
I20260619 16:46:38.188609 9582 ts_manager.cc:194] Registered new tserver with Master: a895265ddbcb4974b8f8ab1497ab7667 (127.3.233.131:33021)
I20260619 16:46:38.189512 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:38.189499 9582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:57643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/test-admin.keytab.
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.207696 4006 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 16:46:38.210599 10049 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:38.209135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.3.233.129:40809)
0619 16:46:38.209413 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:38.209429 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:38.209460 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:46:38.209527 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:38.209538 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:38.209740 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:38.209966 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:38.210393 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33144: BlockingRecv error: recv got EOF from 127.0.0.1:33144 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
W20260619 16:46:38.211026 4006 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc)
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.212085 4006 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.213029 4006 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:38.224205 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.213978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.3.233.129:40809)
0619 16:46:38.214153 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:38.214157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:38.214173 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:38.214313 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:38.214316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:38.214360 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:38.214451 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:38.215222 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.216096 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.216960 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.217511 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.217607 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:38.220494 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:38.220504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:38.220518 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:38.220543 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:38.222361 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.222850 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.222857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.222858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.223000 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.223205 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.223207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.223208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.223452 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:38.223605 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:38.223841 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 16:46:38.223999 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 16:46:38.228546 10049 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:38.227474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.3.233.129:40809)
0619 16:46:38.227670 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:38.227674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:38.227770 (+ 96us) server_negotiation.cc:407] Connection header received
0619 16:46:38.227916 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:38.227919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:38.227959 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:38.228050 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:38.228429 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33172: BlockingRecv error: recv got EOF from 127.0.0.1:33172 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260619 16:46:38.228667 4006 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc)
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.229676 4006 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.230525 4006 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:38.239530 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.231337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.3.233.129:40809)
0619 16:46:38.231484 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:38.231488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:38.231502 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:38.231604 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:38.231607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:38.231643 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:38.231711 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:38.232183 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.232775 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.233569 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.233826 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.233926 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:38.236268 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:38.236272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:38.236274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:38.236297 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:38.237739 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.238144 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.238147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.238148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.238229 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.238482 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.238484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.238486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.238657 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:38.238760 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:38.239232 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 16:46:38.239403 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 16:46:38.242419 10049 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:38.241440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.3.233.129:40809)
0619 16:46:38.241567 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:38.241574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:38.241705 (+ 131us) server_negotiation.cc:407] Connection header received
0619 16:46:38.241798 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:38.241800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:38.241834 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:38.241899 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:38.242303 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33192: BlockingRecv error: recv got EOF from 127.0.0.1:33192 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 16:46:38.242601 4006 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887569513258-4006-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.243871 4006 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:38.244895 4006 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:46:38.254380 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:38.245846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.3.233.129:40809)
0619 16:46:38.245999 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:38.246004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:38.246160 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:46:38.246335 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:38.246338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:38.246380 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:38.246537 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:38.247158 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.247775 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.248642 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:38.248879 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:38.248987 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:38.251490 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:38.251498 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:38.251500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:38.251523 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:38.252817 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.253233 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.253237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.253239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.253318 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:38.253579 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:38.253584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:38.253585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:38.253752 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:38.253885 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:38.254109 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:46:38.254214 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:46:38.255578 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9645
I20260619 16:46:38.262117 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9779
I20260619 16:46:38.267642 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9913
I20260619 16:46:38.272872 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 9551
2026-06-19T16:46:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2267 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[10062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[10062](info): set up 2 sockets
Jun 19 16:46:38 dist-test-slave-cgwx krb5kdc[10062](info): commencing operation
krb5kdc: starting...
W20260619 16:46:39.339426 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:39 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab.
Entry for principal oryx/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab.
I20260619 16:46:39.466471 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35425
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab
--principal=oryx/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:39.577914 10075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:39.578210 10075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:39.578253 10075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:39.582218 10075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:39.582279 10075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:39.582299 10075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:39.582316 10075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:39.582331 10075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:39.587719 10075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab
--principal=oryx/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:39.588757 10075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:39.589737 10075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:39.598551 10081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:39.598654 10083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:39.598573 10080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:39.598959 10075 server_base.cc:1034] running on GCE node
I20260619 16:46:39.599609 10075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:39.600960 10075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:39.602190 10075 hybrid_clock.cc:648] HybridClock initialized: now 1781887599602166 us; error 36 us; skew 500 ppm
Jun 19 16:46:39 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:39.605733 10075 init.cc:377] Logged in from keytab as oryx/127.3.233.190@KRBTEST.COM (short username oryx)
I20260619 16:46:39.607023 10075 webserver.cc:460] Webserver started at http://127.3.233.190:44123/ using document root <none> and password file <none>
I20260619 16:46:39.607704 10075 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:39.607815 10075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:39.608104 10075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:39.610446 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "36153ae3d4194954922333389fcf1a0e"
format_stamp: "Formatted at 2026-06-19 16:46:39 on dist-test-slave-cgwx"
server_key: "10ee2325dc64a281d081f3c088442a95"
server_key_iv: "3fa8efb6f2c959b1de5f2908f0aed7ba"
server_key_version: "encryptionkey@0"
I20260619 16:46:39.611169 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "36153ae3d4194954922333389fcf1a0e"
format_stamp: "Formatted at 2026-06-19 16:46:39 on dist-test-slave-cgwx"
server_key: "10ee2325dc64a281d081f3c088442a95"
server_key_iv: "3fa8efb6f2c959b1de5f2908f0aed7ba"
server_key_version: "encryptionkey@0"
I20260619 16:46:39.615561 10075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:46:39.618418 10090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:39.619942 10075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:39.620121 10075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "36153ae3d4194954922333389fcf1a0e"
format_stamp: "Formatted at 2026-06-19 16:46:39 on dist-test-slave-cgwx"
server_key: "10ee2325dc64a281d081f3c088442a95"
server_key_iv: "3fa8efb6f2c959b1de5f2908f0aed7ba"
server_key_version: "encryptionkey@0"
I20260619 16:46:39.620256 10075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:39.631984 10075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:39.635630 10075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:39.635855 10075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:39.643347 10075 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:35425
I20260619 16:46:39.643656 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:35425 every 8 connection(s)
I20260619 16:46:39.644384 10075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:39.647658 10143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:39.653261 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 10075
I20260619 16:46:39.653445 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:39.653764 4006 external_mini_cluster.cc:1428] Setting key 3ac4090ff64e88abfaabd9eaa26e00bf
I20260619 16:46:39.653837 10143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: Bootstrap starting.
I20260619 16:46:39.656970 10143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:39.657867 10143 log.cc:824] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:39.660604 10143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: No bootstrap required, opened a new log
Jun 19 16:46:39 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:39.665868 10143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } }
I20260619 16:46:39.666168 10143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:39.666266 10143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36153ae3d4194954922333389fcf1a0e, State: Initialized, Role: FOLLOWER
I20260619 16:46:39.666857 10143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } }
I20260619 16:46:39.667073 10143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:39.667183 10143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:39.667414 10143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:39.668512 10143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } }
I20260619 16:46:39.668960 10143 leader_election.cc:302] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36153ae3d4194954922333389fcf1a0e; no voters:
I20260619 16:46:39.669375 10143 leader_election.cc:288] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:39.669446 10148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:39.669658 10148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 1 LEADER]: Becoming Leader. State: Replica: 36153ae3d4194954922333389fcf1a0e, State: Running, Role: LEADER
I20260619 16:46:39.669972 10148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } }
I20260619 16:46:39.670796 10143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:39.671824 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } }
I20260619 16:46:39.671878 10149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36153ae3d4194954922333389fcf1a0e. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } }
I20260619 16:46:39.672127 10149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:39.672202 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:39.672466 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:39.655564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.3.233.190:35425)
0619 16:46:39.656171 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:39.656191 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:39.656234 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:46:39.657176 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:39.657205 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:39.657670 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:39.658060 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:39.659095 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:39.660743 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:39.662106 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:39.662675 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:39.662841 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:39.666201 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:39.666208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:39.666229 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:39.666266 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:39.668620 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:39.669098 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:39.669104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:39.669110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:39.669262 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:39.669610 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:39.669613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:39.669614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:39.670198 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:39.670420 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:39.670734 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 16:46:39.671236 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":200,"threads_started":1}
I20260619 16:46:39.672567 10157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:39.675896 10157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:39.683264 10157 catalog_manager.cc:1347] Generated new cluster ID: 0022aa6f49db4288b0e5d64c7990ba81
I20260619 16:46:39.683357 10157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:39.696501 10157 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:39.697702 10157 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:39.720367 10157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: Generated new TSK 0
I20260619 16:46:39.721202 10157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab.
Entry for principal oryx/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab.
I20260619 16:46:39.754611 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35425
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab
--principal=oryx/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:39.870282 10169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:39.870626 10169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:39.870671 10169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:39.875010 10169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:39.875074 10169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:39.875133 10169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:39.880288 10169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab
--principal=oryx/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:39.881361 10169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:39.882311 10169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:39.892390 10177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:39.892383 10175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:39.892593 10169 server_base.cc:1034] running on GCE node
W20260619 16:46:39.892383 10174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:39.893342 10169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:39.894053 10169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:39.895252 10169 hybrid_clock.cc:648] HybridClock initialized: now 1781887599895221 us; error 42 us; skew 500 ppm
Jun 19 16:46:39 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:39.898262 10169 init.cc:377] Logged in from keytab as oryx/127.3.233.129@KRBTEST.COM (short username oryx)
I20260619 16:46:39.899338 10169 webserver.cc:460] Webserver started at http://127.3.233.129:43517/ using document root <none> and password file <none>
I20260619 16:46:39.899964 10169 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:39.900064 10169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:39.900321 10169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:39.902599 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "4831e1ec601748608f6f319cc3402bc9"
format_stamp: "Formatted at 2026-06-19 16:46:39 on dist-test-slave-cgwx"
server_key: "d824237388ea66009e0148a3add24d1a"
server_key_iv: "9d7c6358214183f69fa202038c3e3d4d"
server_key_version: "encryptionkey@0"
I20260619 16:46:39.903288 10169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "4831e1ec601748608f6f319cc3402bc9"
format_stamp: "Formatted at 2026-06-19 16:46:39 on dist-test-slave-cgwx"
server_key: "d824237388ea66009e0148a3add24d1a"
server_key_iv: "9d7c6358214183f69fa202038c3e3d4d"
server_key_version: "encryptionkey@0"
I20260619 16:46:39.907373 10169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:46:39.909943 10184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:39.911424 10169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:39.911572 10169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "4831e1ec601748608f6f319cc3402bc9"
format_stamp: "Formatted at 2026-06-19 16:46:39 on dist-test-slave-cgwx"
server_key: "d824237388ea66009e0148a3add24d1a"
server_key_iv: "9d7c6358214183f69fa202038c3e3d4d"
server_key_version: "encryptionkey@0"
I20260619 16:46:39.911695 10169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:39.951109 10169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:39.954815 10169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:39.955091 10169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:39.955761 10169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:39.956737 10169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:39.956817 10169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:39.956897 10169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:39.956954 10169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:39.967216 10169 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:39199
I20260619 16:46:39.967236 10297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:39199 every 8 connection(s)
I20260619 16:46:39.968261 10169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:39.977617 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 10169
I20260619 16:46:39.977751 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:39.978080 4006 external_mini_cluster.cc:1428] Setting key f20e0959a2c04c2ab42b628987f86730
Jun 19 16:46:39 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:39.986320 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:39.970181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:46289 (local address 127.3.233.190:35425)
0619 16:46:39.970343 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:39.970347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:39.971498 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 16:46:39.973211 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:39.973215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:39.973270 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:39.973333 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:39.975086 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:39.975896 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:39.977615 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:39.977979 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:39.978063 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:39.981642 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:39.981645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:39.981648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:39.981674 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:39.983688 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:39.984447 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:39.984450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:39.984451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:39.984536 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:39.985030 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:39.985033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:39.985034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:39.985215 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:39.985287 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:39.986085 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 16:46:39.986195 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:46:39.987813 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:39.970637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:46289)
0619 16:46:39.971334 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:39.971392 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:39.972950 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:39.973754 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:39.973763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:39.974241 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:39.974891 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:39.974909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:39.976057 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:39.976060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:39.977459 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:39.977472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:39.978297 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:39.978301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:39.978530 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:39.979332 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:39.979353 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:39.981425 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:39.983828 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:39.983839 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:39.983863 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:39.984321 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:39.984660 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:39.984666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:39.984669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:39.984917 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:39.985327 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:39.985339 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:39.985757 (+ 418us) client_negotiation.cc:769] Sending connection context
0619 16:46:39.986027 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 16:46:39.986476 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":190,"threads_started":1}
I20260619 16:46:39.989431 10298 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35425
WARNING: no policy specified for oryx/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:39.989708 10298 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:39.990222 10298 heartbeater.cc:511] Master 127.3.233.190:35425 requested a full tablet report, sending...
I20260619 16:46:39.991940 10107 ts_manager.cc:194] Registered new tserver with Master: 4831e1ec601748608f6f319cc3402bc9 (127.3.233.129:39199)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:39.993194 10107 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:46289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab.
Entry for principal oryx/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab.
I20260619 16:46:40.011351 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35425
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab
--principal=oryx/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:40.124138 10303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:40.124527 10303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:40.124631 10303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:40.129159 10303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:40.129262 10303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:40.129367 10303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:40.135025 10303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab
--principal=oryx/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:40.136296 10303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:40.137421 10303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:40.148017 10309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.148072 10303 server_base.cc:1034] running on GCE node
W20260619 16:46:40.147979 10308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:40.148013 10311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.148628 10303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:40.149374 10303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:40.150597 10303 hybrid_clock.cc:648] HybridClock initialized: now 1781887600150566 us; error 49 us; skew 500 ppm
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:40.153740 10303 init.cc:377] Logged in from keytab as oryx/127.3.233.130@KRBTEST.COM (short username oryx)
I20260619 16:46:40.154750 10303 webserver.cc:460] Webserver started at http://127.3.233.130:39937/ using document root <none> and password file <none>
I20260619 16:46:40.155440 10303 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:40.155550 10303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:40.155797 10303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:40.157919 10303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "c554cdb022914cc284dde53ab49a744b"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4314bab45b5e05472d4189a865a6d94b"
server_key_iv: "3f7dcb5e01191d1735174f26430d7fb3"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.158528 10303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "c554cdb022914cc284dde53ab49a744b"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4314bab45b5e05472d4189a865a6d94b"
server_key_iv: "3f7dcb5e01191d1735174f26430d7fb3"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.162680 10303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 16:46:40.165548 10318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.167135 10303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:40.167306 10303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "c554cdb022914cc284dde53ab49a744b"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4314bab45b5e05472d4189a865a6d94b"
server_key_iv: "3f7dcb5e01191d1735174f26430d7fb3"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.167454 10303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:40.191736 10303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:40.195760 10303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:40.196003 10303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:40.196687 10303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:40.197656 10303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:40.197736 10303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.197825 10303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:40.197875 10303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.207553 10431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:35289 every 8 connection(s)
I20260619 16:46:40.207554 10303 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:35289
I20260619 16:46:40.208616 10303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:40.218499 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 10303
I20260619 16:46:40.218600 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:40.218871 4006 external_mini_cluster.cc:1428] Setting key 693e909e71742f6d076ba3824f8cf361
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:40.225689 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.210564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:42797 (local address 127.3.233.190:35425)
0619 16:46:40.210699 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.210703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.211661 (+ 958us) server_negotiation.cc:407] Connection header received
0619 16:46:40.213200 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.213203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.213253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.213353 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.214592 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.215265 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.216531 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.216757 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.216870 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.219797 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.219800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.219802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.219829 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.223304 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.223914 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.223919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.223921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.224008 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.224472 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.224475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.224476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.224670 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.224783 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.225384 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.225525 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:46:40.226567 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.210996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:42797)
0619 16:46:40.211503 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.211548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.212755 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.213541 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.213549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.213924 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.214437 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.214450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.215399 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.215403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.216403 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.216412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.216918 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.216922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.217096 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.217734 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.217751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.219600 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.223451 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.223459 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.223479 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.223794 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.224111 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.224114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.224116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.224285 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.224784 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.224794 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.225089 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.225325 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.225703 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260619 16:46:40.227892 10432 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35425
I20260619 16:46:40.228185 10432 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:46:40.228806 10432 heartbeater.cc:511] Master 127.3.233.190:35425 requested a full tablet report, sending...
I20260619 16:46:40.229882 10107 ts_manager.cc:194] Registered new tserver with Master: c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289)
I20260619 16:46:40.230608 10107 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:42797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab.
Entry for principal oryx/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab.
I20260619 16:46:40.246203 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35425
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab
--principal=oryx/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:40.361701 10437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:40.362069 10437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:40.362151 10437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:40.366411 10437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:40.366531 10437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:40.366631 10437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:40.372426 10437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab
--principal=oryx/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:40.373569 10437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:40.374614 10437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:40.384445 10442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.384600 10437 server_base.cc:1034] running on GCE node
W20260619 16:46:40.384447 10443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:40.384763 10445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.385214 10437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:40.385952 10437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:40.387153 10437 hybrid_clock.cc:648] HybridClock initialized: now 1781887600387124 us; error 41 us; skew 500 ppm
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:40.390312 10437 init.cc:377] Logged in from keytab as oryx/127.3.233.131@KRBTEST.COM (short username oryx)
I20260619 16:46:40.391346 10437 webserver.cc:460] Webserver started at http://127.3.233.131:40085/ using document root <none> and password file <none>
I20260619 16:46:40.391923 10437 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:40.392015 10437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:40.392244 10437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:40.394388 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "d7754ab96594477097a0526a28fbf0f2"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "9fff392139baee0b96ec6d90c264eb2a"
server_key_iv: "7685c2637942d6378ce1f157656a934d"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.395053 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "d7754ab96594477097a0526a28fbf0f2"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "9fff392139baee0b96ec6d90c264eb2a"
server_key_iv: "7685c2637942d6378ce1f157656a934d"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.399078 10437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:46:40.401719 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.403378 10437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:40.403535 10437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "d7754ab96594477097a0526a28fbf0f2"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "9fff392139baee0b96ec6d90c264eb2a"
server_key_iv: "7685c2637942d6378ce1f157656a934d"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.403664 10437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:40.415834 10437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:40.419194 10437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:40.419417 10437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:40.420044 10437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:40.420953 10437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:40.421033 10437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.421111 10437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:40.421168 10437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.430878 10437 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:33931
I20260619 16:46:40.430899 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:33931 every 8 connection(s)
I20260619 16:46:40.431986 10437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:40.433645 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 10437
I20260619 16:46:40.433784 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:40.434068 4006 external_mini_cluster.cc:1428] Setting key b5d5130b1390c421bcc647bae84ec100
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:40.446034 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.433963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:34555 (local address 127.3.233.190:35425)
0619 16:46:40.434120 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.434125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.435215 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 16:46:40.436712 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.436715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.436767 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.436842 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.437983 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.438513 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.439729 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.439950 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.440069 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.442575 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.442579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.442581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.442605 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.444158 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.444642 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.444644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.444646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.444717 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.445005 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.445007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.445008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.445181 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.445257 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.445774 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.445873 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:46:40.447064 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.434467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:34555)
0619 16:46:40.435066 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.435128 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.436508 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.437013 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.437020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.437375 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.437849 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.437861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.438635 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.438638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.439597 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.439606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.440102 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.440106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.440271 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.440868 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.440884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.442356 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.444270 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.444276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.444290 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.444552 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.444795 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.444797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.444798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.444926 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.445293 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.445298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.445568 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.445741 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.446016 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":152,"threads_started":1}
I20260619 16:46:40.448267 10566 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35425
I20260619 16:46:40.448588 10566 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:40.449162 10566 heartbeater.cc:511] Master 127.3.233.190:35425 requested a full tablet report, sending...
I20260619 16:46:40.450258 10106 ts_manager.cc:194] Registered new tserver with Master: d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931)
I20260619 16:46:40.450819 10106 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:34555
I20260619 16:46:40.460517 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.233.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.189.keytab.
Entry for principal oryx/127.3.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.189.keytab.
I20260619 16:46:40.490113 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:43481
--webserver_interface=127.3.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.189.keytab
--principal=oryx/127.3.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:35425,127.3.233.189:43481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:40.603971 10571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:40.604257 10571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:40.604297 10571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:40.608570 10571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:40.608642 10571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:40.608662 10571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:40.608680 10571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:40.608695 10571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:40.617157 10571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:35425,127.3.233.189:43481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.189.keytab
--principal=oryx/127.3.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
W20260619 16:46:40.617434 10571 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.233.190:35425,127.3.233.189:43481'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 16:46:40.618629 10571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:40.619781 10571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:40.628420 10579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:40.628409 10577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.628465 10571 server_base.cc:1034] running on GCE node
W20260619 16:46:40.628412 10576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.629323 10571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:40.630470 10571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:40.631644 10571 hybrid_clock.cc:648] HybridClock initialized: now 1781887600631603 us; error 50 us; skew 500 ppm
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:40.634366 10571 init.cc:377] Logged in from keytab as oryx/127.3.233.189@KRBTEST.COM (short username oryx)
I20260619 16:46:40.635356 10571 webserver.cc:460] Webserver started at http://127.3.233.189:45289/ using document root <none> and password file <none>
I20260619 16:46:40.635931 10571 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:40.635987 10571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:40.636150 10571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:40.638170 10571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/instance:
uuid: "594f84c57e764815bafef69f8e073aa3"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4d788f66a13a200f3d59a6a994c81d99"
server_key_iv: "ecc60b3e6776e24654da1fa4af351d40"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.638679 10571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal/instance:
uuid: "594f84c57e764815bafef69f8e073aa3"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4d788f66a13a200f3d59a6a994c81d99"
server_key_iv: "ecc60b3e6776e24654da1fa4af351d40"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.642499 10571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:40.645424 10586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.646796 10571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:40.646955 10571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
uuid: "594f84c57e764815bafef69f8e073aa3"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4d788f66a13a200f3d59a6a994c81d99"
server_key_iv: "ecc60b3e6776e24654da1fa4af351d40"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.647063 10571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:40.658861 10571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:40.662637 10571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:40.662892 10571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:40.670192 10571 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.189:43481
I20260619 16:46:40.670557 10638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.189:43481 every 8 connection(s)
I20260619 16:46:40.671479 10571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
I20260619 16:46:40.674567 10639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:40.676764 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 10571
I20260619 16:46:40.679126 10639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.233.189@KRBTEST.COM
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.189@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:40.691689 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.681265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.3.233.190:35425)
0619 16:46:40.681396 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.681401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.681511 (+ 110us) server_negotiation.cc:407] Connection header received
0619 16:46:40.681657 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.681659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.681704 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.681774 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.682643 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.683466 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.684792 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.685195 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.685299 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.688283 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.688286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.688288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.688314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.689719 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.690203 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.690206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.690208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.690288 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.690608 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.690613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.690615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.690833 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.690970 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.691400 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.691524 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:46:40.692572 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.680879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:39940)
0619 16:46:40.681329 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.681383 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.681557 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.681847 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.681853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.682291 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.682493 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.682506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.683616 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.683619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.684609 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.684619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.685359 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.685369 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.685527 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.686129 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.686145 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.688129 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.689839 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.689847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.689859 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.690099 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.690379 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.690383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.690385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.690506 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.690962 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.690968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.691175 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.691351 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.691723 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":43,"thread_start_us":148,"threads_started":1}
I20260619 16:46:40.693143 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.678489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.3.233.189:43481)
0619 16:46:40.678904 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.678979 (+ 75us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.679022 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:46:40.679983 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.680003 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.680368 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.680731 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.681601 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.683019 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.684026 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.684709 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.684837 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.687820 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.687826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.687839 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.687861 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.689732 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.690208 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.690214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.690216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.690327 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.690640 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.690645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.690647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.691143 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.691353 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.691537 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.692973 (+ 1436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1247,"server-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260619 16:46:40.694139 10639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.189@KRBTEST.COM for oryx/127.3.233.189@KRBTEST.COM
I20260619 16:46:40.702011 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.694610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.0.0.1:55662)
0619 16:46:40.694739 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.694753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.694826 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.695123 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.695126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.695299 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.695391 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.695396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.696102 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.696105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.696615 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.696622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.697032 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.697034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.697106 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.697603 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.697622 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.699010 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.700631 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.700633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.700635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.700835 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.701116 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.701118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.701120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.701200 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.701546 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.701548 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.701621 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.701711 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.701829 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:46:40.702013 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.694687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.3.233.189:43481)
0619 16:46:40.694781 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.694785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.694795 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:46:40.694913 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.694915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.695008 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.695089 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.695491 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.696002 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.696711 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.696925 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.697016 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.699152 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.699155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.699156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.699176 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.700524 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.700933 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.700936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.700938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.701012 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.701278 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.701280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.701281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.701467 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.701539 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.701761 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.701883 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":33}
I20260619 16:46:40.705204 10639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Bootstrap starting.
I20260619 16:46:40.707870 10639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:40.708683 10639 log.cc:824] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:40.710606 10639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: No bootstrap required, opened a new log
I20260619 16:46:40.713573 10639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } }
I20260619 16:46:40.713784 10639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:40.713819 10639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594f84c57e764815bafef69f8e073aa3, State: Initialized, Role: FOLLOWER
I20260619 16:46:40.714201 10639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } }
I20260619 16:46:40.714637 10645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } } }
I20260619 16:46:40.714792 10645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:40.715462 10639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:46:40.720209 10656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:40.720312 10656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:40.722646 10571 master_runner.cc:428] Detected that this master 594f84c57e764815bafef69f8e073aa3 is joining an existing cluster
I20260619 16:46:40.722733 10571 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.233.189:43481
I20260619 16:46:40.734652 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.725375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.3.233.190:35425)
0619 16:46:40.725639 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.725644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.725662 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:40.725710 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.725714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.725780 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.725881 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.726389 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.727148 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.728481 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.728776 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.728889 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.731286 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.731291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.731294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.731325 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.732883 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733347 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.733351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.733353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.733438 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733764 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.733767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.733769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.733968 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.734077 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.734366 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.734481 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260619 16:46:40.734676 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.724968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:39942)
0619 16:46:40.725289 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.725306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.725402 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.725905 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.725907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.726142 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.726255 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.726260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.727253 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.727257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.728337 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.728347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.728905 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.728909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.729070 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.730659 (+ 1589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.730672 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.731130 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.733030 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.733035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.733239 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.733567 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.733576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.733654 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.734066 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.734069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.734194 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.734305 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.734475 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":126,"threads_started":1}
I20260619 16:46:40.735157 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.725498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.3.233.189:43481)
0619 16:46:40.725579 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.725582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.726250 (+ 668us) server_negotiation.cc:407] Connection header received
0619 16:46:40.726463 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.726466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.726509 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.726584 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.727172 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.727797 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.728631 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.729038 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.729147 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.730691 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.730695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.730699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.730727 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.733023 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733517 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.733524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.733527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.733608 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733921 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.733925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.733927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.734140 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.734214 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.734533 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.734648 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":30}
I20260619 16:46:40.735181 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.725021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.0.0.1:55670)
0619 16:46:40.726166 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.726182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.726307 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.726637 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.726640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.726870 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.727048 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.727056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.727911 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.727914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.728511 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.728519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.729158 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.729162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.729262 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.729934 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.729945 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.730524 (+ 579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.733153 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.733158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.733415 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.733712 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.733715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.733717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.733813 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.734277 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.734280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.734371 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.734698 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.734961 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":104,"threads_started":1}
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.190@KRBTEST.COM for oryx/127.3.233.189@KRBTEST.COM
I20260619 16:46:40.750592 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.740276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.3.233.189:43481)
0619 16:46:40.740422 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.740426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.740805 (+ 379us) server_negotiation.cc:407] Connection header received
0619 16:46:40.741007 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.741009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.741054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.741122 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.741971 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.742455 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.743845 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.744058 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.744141 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.747053 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.747067 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.747074 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.747107 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.748730 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.749238 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.749244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.749249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.749332 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.749631 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.749635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.749636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.749831 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.749942 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.750293 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.750422 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:46:40.750633 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.740243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.0.0.1:55674)
0619 16:46:40.740657 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.740718 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.740868 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.741186 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.741193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.741702 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.741849 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.741858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.742560 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.742564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.743725 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.743737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.744174 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.744178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.744298 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.744955 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.744972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.746890 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.748851 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.748862 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.748869 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.749136 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.749445 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.749448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.749450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.749538 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.749944 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.749955 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.750087 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.750240 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.750479 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
I20260619 16:46:40.752523 10106 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.233.189:43481: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "594f84c57e764815bafef69f8e073aa3"
member_type: NON_VOTER
last_known_addr {
host: "127.3.233.189"
port: 43481
}
attrs {
promote: true
}
}
dest_uuid: "36153ae3d4194954922333389fcf1a0e"
cas_config_opid_index: -1
I20260619 16:46:40.753154 10106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } }
I20260619 16:46:40.755088 10668 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 594f84c57e764815bafef69f8e073aa3 (127.3.233.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } }
I20260619 16:46:40.755829 10613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:40.757180 10668 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.233.189:43481
I20260619 16:46:40.757267 10613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Refusing update from remote peer 36153ae3d4194954922333389fcf1a0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 16:46:40.757663 10668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } }
I20260619 16:46:40.757757 10571 master.cc:537] Master@127.3.233.189:43481 shutting down...
WARNING: no policy specified for oryx/127.3.233.188@KRBTEST.COM; defaulting to no policy
I20260619 16:46:40.759209 10668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:40.758311 10667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.188@KRBTEST.COM" created.
W20260619 16:46:40.761868 10094 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 16:46:40.762094 10571 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:40.762389 10571 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:40.762528 10571 tablet_replica.cc:330] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: stopping tablet replica
W20260619 16:46:40.764314 10094 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e -> Peer 594f84c57e764815bafef69f8e073aa3 (127.3.233.189:43481): Couldn't send request to peer 594f84c57e764815bafef69f8e073aa3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:46:40.767630 10571 master.cc:552] Master@127.3.233.189:43481 shutdown complete.
I20260619 16:46:40.767688 10571 master_runner.cc:305] Clearing existing system tablet
I20260619 16:46:40.767943 10571 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:40.768002 10571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:40.769379 10571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:46:40.770334 10673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.770599 10571 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:40.770960 10571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:46:40.771018 10571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
uuid: "594f84c57e764815bafef69f8e073aa3"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4d788f66a13a200f3d59a6a994c81d99"
server_key_iv: "ecc60b3e6776e24654da1fa4af351d40"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.773437 10571 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:40.776823 10571 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:46:40.776922 10571 log.cc:1196] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 16:46:40.777580 10571 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Deleting consensus metadata
I20260619 16:46:40.777776 10571 master_runner.cc:315] Copying system tablet from 127.3.233.190:35425
I20260619 16:46:40.779075 10571 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.233.190:35425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.188.keytab.
Entry for principal oryx/127.3.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.188.keytab.
I20260619 16:46:40.782610 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:37215
--webserver_interface=127.3.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.188.keytab
--principal=oryx/127.3.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:40.788904 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.779762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.3.233.190:35425)
0619 16:46:40.779905 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:40.779910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:40.780156 (+ 246us) server_negotiation.cc:407] Connection header received
0619 16:46:40.780277 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:40.780281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:40.780334 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:40.780490 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:40.780923 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.781576 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.782629 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.783100 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.783234 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.784505 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:40.784514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:40.784516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:40.784542 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:40.786481 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.787181 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.787187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.787192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.787290 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:40.787642 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:40.787648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:40.787653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:40.787939 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:40.788029 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:40.788464 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 16:46:40.788621 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:40.789892 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:40.779690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:39948)
0619 16:46:40.780031 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:40.780051 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:40.780181 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:40.780488 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:40.780491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:40.780671 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:40.780792 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.780797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.781728 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.781736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.782492 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:40.782502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:40.783240 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:40.783244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:40.783349 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:40.783937 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:40.783949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:40.784318 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:40.786631 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.786638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.786643 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.787054 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.787410 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:40.787416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:40.787421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:40.787519 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:40.788113 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:40.788118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:40.788242 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 16:46:40.789379 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0619 16:46:40.789586 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":173,"threads_started":1}
I20260619 16:46:40.790818 10127 tablet_copy_service.cc:140] P 36153ae3d4194954922333389fcf1a0e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 594f84c57e764815bafef69f8e073aa3 ({username='oryx', principal='oryx/127.3.233.189@KRBTEST.COM'} at 127.0.0.1:39948)
I20260619 16:46:40.791069 10127 tablet_copy_service.cc:161] P 36153ae3d4194954922333389fcf1a0e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 594f84c57e764815bafef69f8e073aa3 at {username='oryx', principal='oryx/127.3.233.189@KRBTEST.COM'} at 127.0.0.1:39948: session id = 594f84c57e764815bafef69f8e073aa3-00000000000000000000000000000000
I20260619 16:46:40.793447 10127 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:46:40.795512 10571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:46:40.798054 10571 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: tablet copy: Starting download of 0 data blocks...
I20260619 16:46:40.798261 10571 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: tablet copy: Starting download of 1 WAL segments...
I20260619 16:46:40.800180 10571 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:46:40.802227 10127 tablet_copy_service.cc:342] P 36153ae3d4194954922333389fcf1a0e: Request end of tablet copy session 594f84c57e764815bafef69f8e073aa3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.233.189@KRBTEST.COM'} at 127.0.0.1:39948
I20260619 16:46:40.802361 10127 tablet_copy_service.cc:433] P 36153ae3d4194954922333389fcf1a0e: ending tablet copy session 594f84c57e764815bafef69f8e073aa3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 594f84c57e764815bafef69f8e073aa3
I20260619 16:46:40.810949 10571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:40.813004 10681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:40.813305 10684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:40.815142 10682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.816098 10571 server_base.cc:1034] running on GCE node
I20260619 16:46:40.816303 10571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:40.816634 10571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:40.817795 10571 hybrid_clock.cc:648] HybridClock initialized: now 1781887600817766 us; error 46 us; skew 500 ppm
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:40.819587 10571 init.cc:377] Logged in from keytab as oryx/127.3.233.189@KRBTEST.COM (short username oryx)
I20260619 16:46:40.820137 10571 webserver.cc:460] Webserver started at http://127.3.233.189:45189/ using document root <none> and password file <none>
I20260619 16:46:40.820351 10571 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:40.820410 10571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:40.822005 10571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:40.823200 10691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.823578 10571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:40.823657 10571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
uuid: "594f84c57e764815bafef69f8e073aa3"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4d788f66a13a200f3d59a6a994c81d99"
server_key_iv: "ecc60b3e6776e24654da1fa4af351d40"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.823745 10571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:40.852196 10571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:40.854544 10571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:40.859957 10571 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.189:43481
I20260619 16:46:40.860634 10571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
I20260619 16:46:40.861048 10743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.189:43481 every 8 connection(s)
I20260619 16:46:40.862614 10744 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:46:40.863749 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Bootstrap starting.
I20260619 16:46:40.875298 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:46:40.876148 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: Bootstrap complete.
I20260619 16:46:40.876801 10744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } }
I20260619 16:46:40.876950 10744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 594f84c57e764815bafef69f8e073aa3, State: Initialized, Role: LEARNER
I20260619 16:46:40.877054 10744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } }
I20260619 16:46:40.877426 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } }
I20260619 16:46:40.877545 10744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:40.877635 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:46:40.881526 10758 catalog_manager.cc:1259] Loaded cluster ID: 0022aa6f49db4288b0e5d64c7990ba81
I20260619 16:46:40.881597 10758 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: loading cluster ID for follower catalog manager: success
I20260619 16:46:40.884012 10758 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: acquiring CA information for follower catalog manager: success
I20260619 16:46:40.885586 10758 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 16:46:40.922420 10680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:40.922737 10680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:40.922850 10680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:40.927438 10680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:40.927531 10680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:40.927569 10680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:40.927629 10680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:40.927682 10680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:40.933295 10680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.188.keytab
--principal=oryx/127.3.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:40.934427 10680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:40.935545 10680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:40.944231 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:40.944274 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:40.944242 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:40.944430 10680 server_base.cc:1034] running on GCE node
I20260619 16:46:40.945029 10680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:40.946197 10680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:40.947374 10680 hybrid_clock.cc:648] HybridClock initialized: now 1781887600947341 us; error 58 us; skew 500 ppm
Jun 19 16:46:40 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:40.950162 10680 init.cc:377] Logged in from keytab as oryx/127.3.233.188@KRBTEST.COM (short username oryx)
I20260619 16:46:40.951170 10680 webserver.cc:460] Webserver started at http://127.3.233.188:34913/ using document root <none> and password file <none>
I20260619 16:46:40.951769 10680 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:40.951830 10680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:40.952122 10680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:40.954263 10680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/instance:
uuid: "5089052ac42544bab6bf1f075fa44d0c"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "cea94b042de0693210b38ea0785ac5f1"
server_key_iv: "127dedd17929edb68891ce28e4220f22"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.954964 10680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal/instance:
uuid: "5089052ac42544bab6bf1f075fa44d0c"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "cea94b042de0693210b38ea0785ac5f1"
server_key_iv: "127dedd17929edb68891ce28e4220f22"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.960783 10680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 16:46:40.963922 10773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:40.965225 10680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:46:40.965333 10680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
uuid: "5089052ac42544bab6bf1f075fa44d0c"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "cea94b042de0693210b38ea0785ac5f1"
server_key_iv: "127dedd17929edb68891ce28e4220f22"
server_key_version: "encryptionkey@0"
I20260619 16:46:40.965483 10680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:40.981931 10680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:40.985409 10680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:40.985623 10680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:40.992558 10680 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.188:37215
I20260619 16:46:40.992893 10825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.188:37215 every 8 connection(s)
I20260619 16:46:40.993566 10680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
I20260619 16:46:40.995826 10298 heartbeater.cc:503] Master 127.3.233.190:35425 was elected leader, sending a full tablet report...
I20260619 16:46:40.996819 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:41.000914 10826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:46:41.001492 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 10680
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.188@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.233.188@KRBTEST.COM
I20260619 16:46:41.016520 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.003019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.3.233.190:35425)
0619 16:46:41.003147 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.003154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.003835 (+ 681us) server_negotiation.cc:407] Connection header received
0619 16:46:41.004864 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.004867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.004921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.005053 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.005984 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.006594 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.007796 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.008145 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.008268 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.011540 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.011544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.011546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.011575 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.013611 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.014415 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.014420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.014422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.014548 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.015116 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.015121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.015123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.015390 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.015503 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.016185 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.016344 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:46:41.017308 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.003057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.3.233.188:37215)
0619 16:46:41.003611 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.003628 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.003668 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:46:41.004548 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.004572 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.004828 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.005018 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.005890 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.006787 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.007767 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.008294 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.008467 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.011771 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.011779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.011798 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.011830 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.014277 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.014838 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.014843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.014844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.014915 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.015281 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.015285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.015286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.015743 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.015896 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.016144 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.016431 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":444,"spinlock_wait_cycles":34176,"thread_start_us":175,"threads_started":1}
I20260619 16:46:41.017838 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.003057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:39956)
0619 16:46:41.003664 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.003720 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.004548 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.005130 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.005139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.005550 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.005821 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.005842 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.006743 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.006752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.007661 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.007670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.008332 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.008337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.008467 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.009468 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.009485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.011338 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.013754 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.013762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.013771 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.014285 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.014652 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.014658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.014660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.014866 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.015506 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.015511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.015821 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.016067 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.017688 (+ 1621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":1348,"thread_start_us":147,"threads_started":1}
I20260619 16:46:41.018990 10826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.188@KRBTEST.COM for oryx/127.3.233.189@KRBTEST.COM
I20260619 16:46:41.028124 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.019459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.0.0.1:55686)
0619 16:46:41.019577 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.019595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.019671 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.020100 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.020103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.020238 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.020336 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.020340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.021234 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.021236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.021786 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.021793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.022274 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.022277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.022356 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.023234 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.023268 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.024950 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.026643 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.026646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.026648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.026905 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.027228 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.027231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.027232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.027314 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.027653 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.027655 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.027740 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.027860 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.027979 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:46:41.028132 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.019574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.3.233.189:43481)
0619 16:46:41.019876 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.019879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.019891 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:46:41.019930 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.019933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.019970 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.020048 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.020468 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.021142 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.021906 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.022154 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.022243 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.025143 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.025147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.025149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.025181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.026532 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.027057 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.027060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.027061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.027137 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.027394 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.027396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.027398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.027573 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.027648 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.027892 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.027989 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":123,"threads_started":1}
I20260619 16:46:41.029057 10826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887600, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.188@KRBTEST.COM for oryx/127.3.233.188@KRBTEST.COM
I20260619 16:46:41.036458 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.029511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.0.0.1:44242)
0619 16:46:41.029641 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.029657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.029736 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.030040 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.030043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.030173 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.030250 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.030255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.030959 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.030962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.031477 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.031484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.031887 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.031889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.031945 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.032360 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.032370 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.033540 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.035086 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.035089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.035090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.035273 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.035575 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.035578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.035580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.035665 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.036013 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.036015 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.036093 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.036182 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.036297 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:46:41.036494 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.029591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.3.233.188:37215)
0619 16:46:41.029727 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.029734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.029751 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:41.029814 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.029817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.029905 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.029994 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.030342 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.030832 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.031572 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.031797 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.031872 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.033660 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.033663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.033665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.033685 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.034976 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.035364 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.035367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.035368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.035449 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.035746 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.035748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.035750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.035916 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.035999 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.036231 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.036350 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":40}
I20260619 16:46:41.039788 10826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Bootstrap starting.
I20260619 16:46:41.042071 10826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:41.042836 10826 log.cc:824] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:41.044885 10826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: No bootstrap required, opened a new log
I20260619 16:46:41.048012 10826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } }
I20260619 16:46:41.048218 10826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:41.048251 10826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5089052ac42544bab6bf1f075fa44d0c, State: Initialized, Role: FOLLOWER
I20260619 16:46:41.048715 10826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } }
I20260619 16:46:41.049182 10832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } } }
I20260619 16:46:41.049296 10832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:41.050132 10826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:46:41.055392 10843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:41.055497 10843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:41.059024 10680 master_runner.cc:428] Detected that this master 5089052ac42544bab6bf1f075fa44d0c is joining an existing cluster
I20260619 16:46:41.059098 10680 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.233.188:37215
I20260619 16:46:41.063752 10832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:41.063925 10832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } }
I20260619 16:46:41.064980 10832 leader_election.cc:288] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425), 594f84c57e764815bafef69f8e073aa3 (127.3.233.189:43481)
I20260619 16:46:41.065510 10117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5089052ac42544bab6bf1f075fa44d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36153ae3d4194954922333389fcf1a0e" is_pre_election: true
I20260619 16:46:41.066036 10718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5089052ac42544bab6bf1f075fa44d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594f84c57e764815bafef69f8e073aa3" is_pre_election: true
I20260619 16:46:41.066207 10718 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 LEARNER]: Handling vote request from an unknown peer 5089052ac42544bab6bf1f075fa44d0c
I20260619 16:46:41.066339 10718 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 5089052ac42544bab6bf1f075fa44d0c for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 16:46:41.067236 10776 leader_election.cc:302] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5089052ac42544bab6bf1f075fa44d0c; no voters: 36153ae3d4194954922333389fcf1a0e, 594f84c57e764815bafef69f8e073aa3
I20260619 16:46:41.067723 10832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:41.069115 10832 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 16:46:41.070977 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 10169
I20260619 16:46:41.072243 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.061250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.0.0.1:55690)
0619 16:46:41.062381 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.062393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.062500 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.062879 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.062882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.063225 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.063350 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.063358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.064465 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.064470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.065676 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.065685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.066423 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.066426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.066517 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.067120 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.067135 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.067639 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.069755 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.069757 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.069759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.069967 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.070348 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.070351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.070354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.070499 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.070990 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.070993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.071781 (+ 788us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.071901 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.072050 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":64,"thread_start_us":117,"threads_started":1}
I20260619 16:46:41.072382 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.061349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.3.233.189:43481)
0619 16:46:41.061476 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.061483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.062502 (+ 1019us) server_negotiation.cc:407] Connection header received
0619 16:46:41.062598 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.062601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.062658 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.062742 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.063491 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.064310 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.065800 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.066169 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.066244 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.067793 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.067797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.067800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.067826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.069647 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.070133 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.070137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.070139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.070256 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.070600 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.070605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.070610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.070857 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.070995 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.071984 (+ 989us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.072140 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:46:41.078438 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:39199
--local_ip_for_outbound_sockets=127.3.233.129
--tserver_master_addrs=127.3.233.190:35425
--webserver_port=43517
--webserver_interface=127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab
--principal=oryx/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:41.092434 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.061741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.3.233.188:37215)
0619 16:46:41.062232 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.062236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.062743 (+ 507us) server_negotiation.cc:407] Connection header received
0619 16:46:41.062882 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.062885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.062917 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.062993 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.063571 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.064222 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.067725 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.068048 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.068122 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.088561 (+ 20439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.088565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.088567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.088588 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.090574 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.091160 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.091163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.091165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.091240 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.091550 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.091553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.091555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.091770 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.091847 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.092134 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.092277 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260619 16:46:41.092430 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.061621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.3.233.190:35425)
0619 16:46:41.061879 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.061883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.061896 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:41.061945 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.061948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.061984 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.062077 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.062627 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.063487 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.066244 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.066579 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.066651 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.088581 (+ 21930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.088585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.088588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.088620 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.090660 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.091175 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.091178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.091180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.091251 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.091561 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.091564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.091566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.091770 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.091843 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.092151 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.092257 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260619 16:46:41.093106 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.061673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.0.0.1:44250)
0619 16:46:41.062681 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.062693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.062799 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.063102 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.063104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.063340 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.063455 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.063461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.064509 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.064512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.067605 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.067614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.068329 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.068332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.068432 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.087743 (+ 19311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.087759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.088373 (+ 614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.090709 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.090714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.090717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.091053 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.091350 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.091352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.091354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.091458 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.091891 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.091895 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.091976 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.092788 (+ 812us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.092923 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":88,"spinlock_wait_cycles":60672,"thread_start_us":106,"threads_started":1}
I20260619 16:46:41.093502 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.061149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:39964)
0619 16:46:41.061640 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.061657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.061758 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.062096 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.062099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.062337 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.062498 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.062505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.065345 (+ 2840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.065348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.066108 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.066117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.066796 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.066799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.066887 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.087654 (+ 20767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.087678 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.088368 (+ 690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.090772 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.090775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.090777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.091070 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.091354 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.091356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.091359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.091459 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.091906 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.091908 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.091977 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.092986 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.093100 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"spinlock_wait_cycles":6656,"thread_start_us":111,"threads_started":1}
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887599, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.190@KRBTEST.COM for oryx/127.3.233.188@KRBTEST.COM
I20260619 16:46:41.105468 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.097511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.0.0.1:44252)
0619 16:46:41.097674 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.097689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.097774 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.098051 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.098054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.098244 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.098330 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.098335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.099134 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.099140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.099683 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.099691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.100154 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.100158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.100249 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.100765 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.100779 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.102067 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.103969 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.103974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.103977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.104179 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.104463 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.104467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.104469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.104555 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.104938 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.104941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.105042 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.105144 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.105302 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 16:46:41.106335 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.097554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.3.233.188:37215)
0619 16:46:41.097660 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.097664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.097731 (+ 67us) server_negotiation.cc:407] Connection header received
0619 16:46:41.097894 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.097897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.097951 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.098035 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.098445 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.099012 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.099806 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.100043 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.100151 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.102212 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.102217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.102220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.102250 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.103848 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.104276 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.104281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.104283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.104365 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.104649 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.104653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.104654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.104837 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.104942 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.105203 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.106219 (+ 1016us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:46:41.107288 10106 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.233.188:37215: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c"
member_type: NON_VOTER
last_known_addr {
host: "127.3.233.188"
port: 37215
}
attrs {
promote: true
}
}
dest_uuid: "36153ae3d4194954922333389fcf1a0e"
cas_config_opid_index: 5
I20260619 16:46:41.107853 10106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } }
I20260619 16:46:41.108953 10667 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.110158 10667 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.233.188:37215
I20260619 16:46:41.110343 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.110460 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:41.110836 10680 master.cc:537] Master@127.3.233.188:37215 shutting down...
W20260619 16:46:41.112537 10094 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e -> Peer 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188:37215): Couldn't send request to peer 5089052ac42544bab6bf1f075fa44d0c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:46:41.115990 10680 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:41.116290 10680 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:41.116381 10680 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: stopping tablet replica
I20260619 16:46:41.117480 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.111108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.3.233.189:43481)
0619 16:46:41.111820 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.111824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.112934 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 16:46:41.113154 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.113157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.113225 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.113329 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:41.113585 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.114301 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.115847 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.116499 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.116581 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.116729 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.117267 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.117353 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622}
I20260619 16:46:41.118156 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.110837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.0.0.1:55706)
0619 16:46:41.112861 (+ 2024us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.112876 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.113014 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.113352 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.113355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.113371 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:41.113474 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.113484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.114443 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.114448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.115698 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.115709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.116971 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.116974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.117078 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.117140 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.117834 (+ 694us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.118008 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1779}
I20260619 16:46:41.118865 10718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 LEARNER]: Refusing update from remote peer 36153ae3d4194954922333389fcf1a0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:46:41.119544 10668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 16:46:41.121503 10718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.123699 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36153ae3d4194954922333389fcf1a0e. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.123786 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:46:41.123910 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: NON_VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: true } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.123996 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:46:41.125130 10668 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: attempting to promote NON_VOTER 594f84c57e764815bafef69f8e073aa3 to VOTER
I20260619 16:46:41.125535 10668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } }
I20260619 16:46:41.126801 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.126550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.0.0.1:44256)
0619 16:46:41.126649 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.126723 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:37215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 16:46:41.127018 10718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 LEARNER]: Refusing update from remote peer 36153ae3d4194954922333389fcf1a0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 16:46:41.127444 10855 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 16:46:41.128042 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.127702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.0.0.1:44266)
0619 16:46:41.127928 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.127980 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:37215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260619 16:46:41.128154 10094 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e -> Peer 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188:37215): Couldn't send request to peer 5089052ac42544bab6bf1f075fa44d0c. Status: Network error: Client connection negotiation failed: client connection to 127.3.233.188:37215: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:46:41.129073 10667 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 594f84c57e764815bafef69f8e073aa3 (127.3.233.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.129606 10718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 594f84c57e764815bafef69f8e073aa3 (127.3.233.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.130648 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.130769 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:41.130906 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.131011 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:41.131372 10854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.131505 10854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:41.135629 10680 master.cc:552] Master@127.3.233.188:37215 shutdown complete.
I20260619 16:46:41.135704 10680 master_runner.cc:305] Clearing existing system tablet
I20260619 16:46:41.135932 10680 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:41.136003 10680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:41.137076 10680 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 16:46:41.137884 10858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.138125 10680 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:41.138341 10680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:41.138381 10680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
uuid: "5089052ac42544bab6bf1f075fa44d0c"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "cea94b042de0693210b38ea0785ac5f1"
server_key_iv: "127dedd17929edb68891ce28e4220f22"
server_key_version: "encryptionkey@0"
I20260619 16:46:41.140228 10680 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:41.142019 10680 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:46:41.142123 10680 log.cc:1196] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 16:46:41.142356 10680 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Deleting consensus metadata
I20260619 16:46:41.142616 10680 master_runner.cc:315] Copying system tablet from 127.3.233.190:35425
I20260619 16:46:41.143780 10680 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.233.190:35425
I20260619 16:46:41.151742 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.144386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.3.233.190:35425)
0619 16:46:41.144531 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.144536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.144784 (+ 248us) server_negotiation.cc:407] Connection header received
0619 16:46:41.144923 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.144926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.144979 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.145065 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.145571 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.146099 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.146917 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.147182 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.147292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.148397 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.148401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.148404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.148443 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.149875 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.150315 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.150320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.150322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.150439 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.150736 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.150739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.150741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.150977 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.151090 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.151428 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.151565 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:46:41.151770 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.144329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:39976)
0619 16:46:41.144691 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.144703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.144831 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.145081 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.145084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.145326 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.145461 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.145467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.146206 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.146212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.146802 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.146809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.147296 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.147300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.147389 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.147925 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.147944 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.148253 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.149990 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.149996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.149998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.150197 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.150535 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.150539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.150540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.150640 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.151079 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.151082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.151187 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.151328 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.151564 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":83,"threads_started":1}
I20260619 16:46:41.152462 10127 tablet_copy_service.cc:140] P 36153ae3d4194954922333389fcf1a0e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5089052ac42544bab6bf1f075fa44d0c ({username='oryx', principal='oryx/127.3.233.188@KRBTEST.COM'} at 127.0.0.1:39976)
I20260619 16:46:41.152590 10127 tablet_copy_service.cc:161] P 36153ae3d4194954922333389fcf1a0e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5089052ac42544bab6bf1f075fa44d0c at {username='oryx', principal='oryx/127.3.233.188@KRBTEST.COM'} at 127.0.0.1:39976: session id = 5089052ac42544bab6bf1f075fa44d0c-00000000000000000000000000000000
I20260619 16:46:41.153440 10127 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:46:41.154606 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:46:41.156591 10680 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: tablet copy: Starting download of 0 data blocks...
I20260619 16:46:41.156785 10680 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: tablet copy: Starting download of 1 WAL segments...
I20260619 16:46:41.158288 10680 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:46:41.160193 10127 tablet_copy_service.cc:342] P 36153ae3d4194954922333389fcf1a0e: Request end of tablet copy session 5089052ac42544bab6bf1f075fa44d0c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.233.188@KRBTEST.COM'} at 127.0.0.1:39976
I20260619 16:46:41.160295 10127 tablet_copy_service.cc:433] P 36153ae3d4194954922333389fcf1a0e: ending tablet copy session 5089052ac42544bab6bf1f075fa44d0c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5089052ac42544bab6bf1f075fa44d0c
I20260619 16:46:41.167265 10680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:41.168938 10865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:41.168993 10868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.169179 10680 server_base.cc:1034] running on GCE node
W20260619 16:46:41.168993 10866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.169608 10680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:41.169910 10680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:41.171037 10680 hybrid_clock.cc:648] HybridClock initialized: now 1781887601171024 us; error 28 us; skew 500 ppm
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:41.172807 10680 init.cc:377] Logged in from keytab as oryx/127.3.233.188@KRBTEST.COM (short username oryx)
I20260619 16:46:41.173290 10680 webserver.cc:460] Webserver started at http://127.3.233.188:34975/ using document root <none> and password file <none>
I20260619 16:46:41.173478 10680 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:41.173532 10680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:41.174638 10680 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:41.175632 10875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.175858 10680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:46:41.175949 10680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
uuid: "5089052ac42544bab6bf1f075fa44d0c"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "cea94b042de0693210b38ea0785ac5f1"
server_key_iv: "127dedd17929edb68891ce28e4220f22"
server_key_version: "encryptionkey@0"
I20260619 16:46:41.176047 10680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:41.185662 10680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:41.187700 10680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:41.192451 10680 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.188:37215
I20260619 16:46:41.192833 10927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.188:37215 every 8 connection(s)
I20260619 16:46:41.193142 10680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
I20260619 16:46:41.194631 10928 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:46:41.195657 10928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Bootstrap starting.
I20260619 16:46:41.204763 10928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:46:41.205109 10928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Bootstrap complete.
I20260619 16:46:41.205651 10928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } }
I20260619 16:46:41.205796 10928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5089052ac42544bab6bf1f075fa44d0c, State: Initialized, Role: LEARNER
I20260619 16:46:41.205916 10928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } }
I20260619 16:46:41.206269 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.206432 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: This master's current role is: LEARNER
I20260619 16:46:41.206465 10928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:41.213371 10942 catalog_manager.cc:1259] Loaded cluster ID: 0022aa6f49db4288b0e5d64c7990ba81
I20260619 16:46:41.213469 10942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: loading cluster ID for follower catalog manager: success
I20260619 16:46:41.215263 10942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: acquiring CA information for follower catalog manager: success
I20260619 16:46:41.216411 10942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:46:41.232723 10432 heartbeater.cc:503] Master 127.3.233.190:35425 was elected leader, sending a full tablet report...
W20260619 16:46:41.240887 10852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:41.241261 10852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:41.241323 10852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:41.245998 10852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:41.246069 10852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:41.246166 10852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:41.251705 10852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab
--principal=oryx/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:39199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=43517
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:41.252863 10852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:41.254024 10852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:41.263976 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.264042 10852 server_base.cc:1034] running on GCE node
W20260619 16:46:41.264045 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:41.263957 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.264669 10852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:41.265303 10852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:41.266498 10852 hybrid_clock.cc:648] HybridClock initialized: now 1781887601266480 us; error 30 us; skew 500 ppm
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:41.269383 10852 init.cc:377] Logged in from keytab as oryx/127.3.233.129@KRBTEST.COM (short username oryx)
I20260619 16:46:41.270198 10852 webserver.cc:460] Webserver started at http://127.3.233.129:43517/ using document root <none> and password file <none>
I20260619 16:46:41.270716 10852 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:41.270785 10852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:41.275318 10852 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:46:41.277638 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.279207 10852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:41.279352 10852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "4831e1ec601748608f6f319cc3402bc9"
format_stamp: "Formatted at 2026-06-19 16:46:39 on dist-test-slave-cgwx"
server_key: "d824237388ea66009e0148a3add24d1a"
server_key_iv: "9d7c6358214183f69fa202038c3e3d4d"
server_key_version: "encryptionkey@0"
I20260619 16:46:41.279891 10852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:41.290395 10852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:41.293885 10852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:41.294097 10852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:41.294719 10852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:41.295626 10852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:41.295671 10852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.295711 10852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:41.295768 10852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.304100 10852 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:39199
I20260619 16:46:41.304176 11070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:39199 every 8 connection(s)
I20260619 16:46:41.305150 10852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:41.312860 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 10852
I20260619 16:46:41.313064 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 10303
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.189@KRBTEST.COM
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.188@KRBTEST.COM
I20260619 16:46:41.320479 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:35289
--local_ip_for_outbound_sockets=127.3.233.130
--tserver_master_addrs=127.3.233.190:35425
--webserver_port=39937
--webserver_interface=127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab
--principal=oryx/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:41.322845 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.307069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:49897 (local address 127.3.233.190:35425)
0619 16:46:41.307199 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.307205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.308173 (+ 968us) server_negotiation.cc:407] Connection header received
0619 16:46:41.309459 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.309463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.309526 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.309582 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.311042 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.311604 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.313800 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.314131 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.314206 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.318129 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.318133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.318136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.318165 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.320834 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321561 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.321567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.321573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.321656 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321962 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.321968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.321970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.322164 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.322244 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.322545 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.322659 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:46:41.322896 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.306996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:34737 (local address 127.3.233.189:43481)
0619 16:46:41.307130 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.307135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.308170 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 16:46:41.309447 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.309451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.309517 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.309611 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.311093 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.311864 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.313917 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.314243 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.314322 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.318609 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.318612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.318615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.318642 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.320207 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.320894 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.320897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.320899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.320988 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321406 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.321412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.321416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.321670 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.321768 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.322538 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.322677 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:46:41.323998 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.308735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:45131 (local address 127.3.233.188:37215)
0619 16:46:41.309068 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.309072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.309085 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:41.309548 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.309554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.309609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.309688 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.311056 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.312120 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.315834 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.316221 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.316344 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.318658 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.318662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.318666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.318698 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.320695 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321192 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.321196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.321197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.321300 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321616 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.321622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.321624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.321850 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.321940 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.323678 (+ 1738us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.323806 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260619 16:46:41.324221 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.308593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.3.233.129:45131)
0619 16:46:41.308849 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.308863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.309225 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.309795 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.309798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.310260 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.310868 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.310881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.312228 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.312233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.315677 (+ 3444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.315691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.316351 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.316354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.316463 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.316979 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.316990 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.318490 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.320828 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.320830 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.320832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.321090 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.321419 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.321423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.321517 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.321981 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.321983 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.322220 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.322457 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.322814 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1093,"spinlock_wait_cycles":37120,"thread_start_us":79,"threads_started":1}
I20260619 16:46:41.323993 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.307480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:49897)
0619 16:46:41.308002 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.308056 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.309225 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.309790 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.309798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.310260 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.310868 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.310881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.311760 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.311770 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.313646 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.313661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.314433 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.314438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.314728 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.315573 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.315597 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.317952 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.321198 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.321202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.321450 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.321764 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.321768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.321841 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.322300 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.322302 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.322368 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.323377 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.323521 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":476,"spinlock_wait_cycles":21248,"thread_start_us":125,"threads_started":1}
I20260619 16:46:41.324373 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.307746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.3.233.129:34737)
0619 16:46:41.308008 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.308057 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.309225 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.309789 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.309797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.310255 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.310866 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.310880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.312035 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.312041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.313793 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.313801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.314475 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.314478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.314803 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.315526 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.315545 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.318453 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.320345 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.320354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.320375 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.320777 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.321096 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.321102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.321107 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.321288 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.321847 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.321855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.322220 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.323142 (+ 922us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.323291 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":873,"spinlock_wait_cycles":24320,"thread_start_us":136,"threads_started":1}
I20260619 16:46:41.325726 11071 heartbeater.cc:347] Connected to a master server at 127.3.233.188:37215
I20260619 16:46:41.325987 11071 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.326251 11073 heartbeater.cc:347] Connected to a master server at 127.3.233.189:43481
I20260619 16:46:41.326339 11073 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.326623 11072 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35425
I20260619 16:46:41.326589 11073 heartbeater.cc:511] Master 127.3.233.189:43481 requested a full tablet report, sending...
I20260619 16:46:41.326593 11071 heartbeater.cc:511] Master 127.3.233.188:37215 requested a full tablet report, sending...
I20260619 16:46:41.326722 11072 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.326895 11072 heartbeater.cc:511] Master 127.3.233.190:35425 requested a full tablet report, sending...
I20260619 16:46:41.327963 10106 ts_manager.cc:194] Re-registered known tserver with Master: 4831e1ec601748608f6f319cc3402bc9 (127.3.233.129:39199)
I20260619 16:46:41.328178 10708 ts_manager.cc:194] Registered new tserver with Master: 4831e1ec601748608f6f319cc3402bc9 (127.3.233.129:39199)
I20260619 16:46:41.328676 10106 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:49897
I20260619 16:46:41.329562 10892 ts_manager.cc:194] Registered new tserver with Master: 4831e1ec601748608f6f319cc3402bc9 (127.3.233.129:39199)
W20260619 16:46:41.440785 11079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:41.441213 11079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:41.441296 11079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:41.445523 11079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:41.445618 11079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:41.445719 11079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:41.451241 11079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab
--principal=oryx/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:35289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=39937
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:41.452359 11079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:41.453035 10566 heartbeater.cc:503] Master 127.3.233.190:35425 was elected leader, sending a full tablet report...
I20260619 16:46:41.453693 11079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:41.464321 11085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.464429 11079 server_base.cc:1034] running on GCE node
W20260619 16:46:41.464345 11084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:41.464345 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.465072 11079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:41.465731 11079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:41.466917 11079 hybrid_clock.cc:648] HybridClock initialized: now 1781887601466893 us; error 36 us; skew 500 ppm
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:41.469956 11079 init.cc:377] Logged in from keytab as oryx/127.3.233.130@KRBTEST.COM (short username oryx)
I20260619 16:46:41.471025 11079 webserver.cc:460] Webserver started at http://127.3.233.130:39937/ using document root <none> and password file <none>
I20260619 16:46:41.471738 11079 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:41.471846 11079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:41.475878 11079 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:46:41.478124 11094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.479615 11079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:41.479789 11079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "c554cdb022914cc284dde53ab49a744b"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "4314bab45b5e05472d4189a865a6d94b"
server_key_iv: "3f7dcb5e01191d1735174f26430d7fb3"
server_key_version: "encryptionkey@0"
I20260619 16:46:41.480322 11079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:41.490831 11079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:41.495170 11079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:41.495384 11079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:41.496253 11079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:41.497486 11079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:41.497570 11079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.497639 11079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:41.497674 11079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.510907 11079 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:35289
I20260619 16:46:41.510978 11207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:35289 every 8 connection(s)
I20260619 16:46:41.512485 11079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:41.520844 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 11079
I20260619 16:46:41.521027 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 10437
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.189@KRBTEST.COM
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.188@KRBTEST.COM
I20260619 16:46:41.530696 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:33931
--local_ip_for_outbound_sockets=127.3.233.131
--tserver_master_addrs=127.3.233.190:35425
--webserver_port=40085
--webserver_interface=127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab
--principal=oryx/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:41.531876 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.516266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:32951 (local address 127.3.233.189:43481)
0619 16:46:41.516402 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.516409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.516734 (+ 325us) server_negotiation.cc:407] Connection header received
0619 16:46:41.517483 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.517488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.517566 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.517628 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.519707 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.520644 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.522331 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.522665 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.522736 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.526829 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.526836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.526841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.526878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.528959 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.529808 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.529814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.529818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.529922 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.530330 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.530334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.530336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.530572 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.530656 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.531488 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.531650 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:41.531883 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.514555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:35913 (local address 127.3.233.188:37215)
0619 16:46:41.514727 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.514734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.515735 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 16:46:41.517423 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.517427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.517488 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.517542 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.519679 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.520515 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.522496 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.523002 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.523093 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.527295 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.527301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.527307 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.527339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.529251 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.529829 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.529836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.529841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.529944 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.530344 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.530347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.530349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.530574 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.530660 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.531476 (+ 816us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.531633 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:46:41.532573 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.521264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.0.0.1:44270)
0619 16:46:41.521575 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.521590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.521664 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.522492 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.522495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.522512 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:41.522595 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.522601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.523723 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.523730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.525304 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.525319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.532155 (+ 6836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.532159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.532262 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.532287 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.532345 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.532427 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260619 16:46:41.533231 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.521854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.3.233.188:37215)
0619 16:46:41.522229 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.522235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.522249 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:41.522296 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.522302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.522365 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.522692 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:41.522807 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.523526 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.530119 (+ 6593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.531043 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.532839 (+ 1796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.533011 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.533049 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.533122 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":71,"threads_started":1}
I20260619 16:46:41.533692 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.516117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.3.233.130:32951)
0619 16:46:41.516649 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.516662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.517186 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.517959 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.517971 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.518659 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.519505 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.519524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.520934 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.520937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.522198 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.522207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.523013 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.523019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.523210 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.524207 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.524235 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.526601 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.529130 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.529138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.529159 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.529664 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.530036 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.530039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.530041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.530226 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.530741 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.530744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.531184 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.532110 (+ 926us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.532560 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":110,"spinlock_wait_cycles":6784,"thread_start_us":72,"threads_started":1}
I20260619 16:46:41.533689 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.515057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.3.233.130:35913)
0619 16:46:41.515571 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.515626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.517171 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.517959 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.517969 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.518658 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.519501 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.519518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.520704 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.520706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.522347 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.522361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.523250 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.523256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.523394 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.525501 (+ 2107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.525522 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.527131 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.529392 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.529398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.529403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.529708 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.530055 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.530061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.530066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.530235 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.530730 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.530737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.531182 (+ 445us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.532112 (+ 930us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.532655 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":852,"spinlock_wait_cycles":4992,"thread_start_us":135,"threads_started":1}
I20260619 16:46:41.533910 10902 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 16:46:41.534368 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36153ae3d4194954922333389fcf1a0e. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.534502 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: This master's current role is: LEARNER
I20260619 16:46:41.535156 11208 heartbeater.cc:347] Connected to a master server at 127.3.233.188:37215
I20260619 16:46:41.535404 11208 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.536017 11208 heartbeater.cc:511] Master 127.3.233.188:37215 requested a full tablet report, sending...
I20260619 16:46:41.536586 11210 heartbeater.cc:347] Connected to a master server at 127.3.233.189:43481
I20260619 16:46:41.536667 11210 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.536860 11210 heartbeater.cc:511] Master 127.3.233.189:43481 requested a full tablet report, sending...
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:41.537367 10892 ts_manager.cc:194] Registered new tserver with Master: c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289)
I20260619 16:46:41.537688 10708 ts_manager.cc:194] Registered new tserver with Master: c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289)
I20260619 16:46:41.541564 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.514578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:32991 (local address 127.3.233.190:35425)
0619 16:46:41.514706 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.514711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.516544 (+ 1833us) server_negotiation.cc:407] Connection header received
0619 16:46:41.517432 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.517435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.517486 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.517542 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.519679 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.520515 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.534846 (+ 14331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.535220 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.535297 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.537840 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.537844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.537847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.537875 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.539836 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.540318 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.540321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.540324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.540406 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.540729 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.540733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.540734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.540936 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.540995 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.541289 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.541412 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:46:41.541602 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.515056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:32991)
0619 16:46:41.516437 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.516451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.517171 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.517959 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.517969 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.518632 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.519498 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.519518 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.532881 (+ 13363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.532885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.534706 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.534717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.535490 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.535494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.535578 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.536180 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.536193 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.537688 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.539972 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.539975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.539976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.540181 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.540519 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.540522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.540524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.540623 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.541050 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.541053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.541122 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.541221 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.541377 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"mutex_wait_us":149,"spinlock_wait_cycles":30464,"thread_start_us":134,"threads_started":1}
I20260619 16:46:41.542420 11209 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35425
I20260619 16:46:41.542536 11209 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.542717 11209 heartbeater.cc:511] Master 127.3.233.190:35425 requested a full tablet report, sending...
I20260619 16:46:41.543448 10106 ts_manager.cc:194] Re-registered known tserver with Master: c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289)
I20260619 16:46:41.544009 10106 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:32991
W20260619 16:46:41.658209 11216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:41.658480 11216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:41.658525 11216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:41.663782 11216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:41.663862 11216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:41.663955 11216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:41.669670 11216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab
--principal=oryx/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:33931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=40085
--tserver_master_addrs=127.3.233.190:35425,127.3.233.189:43481,127.3.233.188:37215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:41.670811 11216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:41.672004 11216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:41.684324 11221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:41.684324 11222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.684780 11216 server_base.cc:1034] running on GCE node
W20260619 16:46:41.684454 11224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:41.685523 11216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:41.686482 11216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:41.687708 11216 hybrid_clock.cc:648] HybridClock initialized: now 1781887601687674 us; error 42 us; skew 500 ppm
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:41.691465 11216 init.cc:377] Logged in from keytab as oryx/127.3.233.131@KRBTEST.COM (short username oryx)
I20260619 16:46:41.692411 11216 webserver.cc:460] Webserver started at http://127.3.233.131:40085/ using document root <none> and password file <none>
I20260619 16:46:41.692999 11216 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:41.693095 11216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:41.697403 11216 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:46:41.699769 11231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.701153 11216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:46:41.701316 11216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "d7754ab96594477097a0526a28fbf0f2"
format_stamp: "Formatted at 2026-06-19 16:46:40 on dist-test-slave-cgwx"
server_key: "9fff392139baee0b96ec6d90c264eb2a"
server_key_iv: "7685c2637942d6378ce1f157656a934d"
server_key_version: "encryptionkey@0"
I20260619 16:46:41.701918 11216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:41.716240 11216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:41.719519 11216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:41.719726 11216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:41.720343 11216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:41.721215 11216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:41.721287 11216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.721355 11216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:41.721406 11216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:41.731164 11216 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:33931
I20260619 16:46:41.731244 11344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:33931 every 8 connection(s)
I20260619 16:46:41.732280 11216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:41.735553 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 11216
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.188@KRBTEST.COM
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.189@KRBTEST.COM
Jun 19 16:46:41 dist-test-slave-cgwx krb5kdc[10062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887601, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:41.750116 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.736134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:52589 (local address 127.3.233.188:37215)
0619 16:46:41.736287 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.736291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.736307 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:41.736627 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.736632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.736694 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.736751 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.738916 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.739737 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.741736 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.742095 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.742169 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.745571 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.745576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.745578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.745605 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.747507 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.748186 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.748191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.748193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.748278 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.748699 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.748705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.748707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.748983 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.749070 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.749770 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.749914 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":81}
I20260619 16:46:41.750514 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.738548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.3.233.190:35425)
0619 16:46:41.739446 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.739450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.739807 (+ 357us) server_negotiation.cc:407] Connection header received
0619 16:46:41.739855 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.739858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.739921 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.740003 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.740920 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.741674 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.742945 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.743321 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.743440 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.746542 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.746549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.746558 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.746589 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.748419 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.748909 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.748913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.748914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.749001 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.749284 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.749286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.749288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.749496 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.749621 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.750221 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.750349 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":823,"thread_start_us":72,"threads_started":1}
I20260619 16:46:41.751993 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.735821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.3.233.131:52589)
0619 16:46:41.736108 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.736123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.736379 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.737149 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.737159 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.737904 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.738744 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.738764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.739900 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.739906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.741603 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.741613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.742380 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.742385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.742606 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.743318 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.743339 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.745363 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.747643 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.747651 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.747671 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.748060 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.748394 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.748397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.748399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.748589 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.749131 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.749140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.749487 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.750337 (+ 850us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.750699 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":131,"spinlock_wait_cycles":32256,"thread_start_us":99,"threads_started":1}
I20260619 16:46:41.753585 11345 heartbeater.cc:347] Connected to a master server at 127.3.233.188:37215
I20260619 16:46:41.753834 11345 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.754349 11345 heartbeater.cc:511] Master 127.3.233.188:37215 requested a full tablet report, sending...
I20260619 16:46:41.754782 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.734218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:41661 (local address 127.3.233.189:43481)
0619 16:46:41.734343 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.734348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.735338 (+ 990us) server_negotiation.cc:407] Connection header received
0619 16:46:41.736610 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.736615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.736668 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.736725 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.738899 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.739702 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.744004 (+ 4302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.744376 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.744485 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.749993 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.749997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.749999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.750033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.751757 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.752608 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.752613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.752614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.752867 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.753173 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.753176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.753177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.753403 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.753507 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.754550 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.754675 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:46:41.755323 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.734645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.3.233.131:41661)
0619 16:46:41.735174 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.735230 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.736363 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.737148 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.737159 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.737965 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.738744 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.738764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.739862 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.739865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.743881 (+ 4016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.743889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.744641 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.744647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.744745 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.745266 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.745285 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.748677 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.751884 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.751887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.751889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.752141 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.752979 (+ 838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.752982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.752984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.753074 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.753564 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.753568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.753672 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.754939 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.755116 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":2019,"spinlock_wait_cycles":5504,"thread_start_us":223,"threads_started":1}
I20260619 16:46:41.755700 10892 ts_manager.cc:194] Registered new tserver with Master: d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931)
I20260619 16:46:41.756762 11347 heartbeater.cc:347] Connected to a master server at 127.3.233.189:43481
I20260619 16:46:41.756851 11347 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.757035 11347 heartbeater.cc:511] Master 127.3.233.189:43481 requested a full tablet report, sending...
I20260619 16:46:41.757535 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.737764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.3.233.189:43481)
0619 16:46:41.739032 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.739036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.739241 (+ 205us) server_negotiation.cc:407] Connection header received
0619 16:46:41.739318 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.739321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.739367 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.740425 (+ 1058us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.740534 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.741260 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.742406 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.742734 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.743251 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.744466 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.744473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.744478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.744506 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.752429 (+ 7923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.756205 (+ 3776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.756209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.756210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.756309 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.756633 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.756639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.756643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.756889 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.757039 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.757256 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.757374 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":89,"threads_started":1}
I20260619 16:46:41.757795 10708 ts_manager.cc:194] Registered new tserver with Master: d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931)
I20260619 16:46:41.759476 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.734646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:60603)
0619 16:46:41.735173 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.735230 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.736535 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.737148 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.737159 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.737905 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:41.738764 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.738774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.741177 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.741180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.742059 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.742068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.744050 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.744054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.744142 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.744671 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:41.744684 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:41.748309 (+ 3625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:41.755470 (+ 7161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.755474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.755478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.755733 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.758270 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:41.758273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:41.758275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:41.758366 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:41.758963 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:41.758966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:41.759061 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.759158 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.759298 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":104,"spinlock_wait_cycles":42752,"thread_start_us":208,"threads_started":1}
I20260619 16:46:41.756276 10106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:46:41.759953 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.734187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:60603 (local address 127.3.233.190:35425)
0619 16:46:41.734358 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.734365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.736464 (+ 2099us) server_negotiation.cc:407] Connection header received
0619 16:46:41.736639 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.736642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.736692 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.736744 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.740265 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.741049 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.743451 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.743748 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.743803 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.748822 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.748826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.748828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.748852 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.753482 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.756578 (+ 3096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.756582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.756584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.756665 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.758593 (+ 1928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.758596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.758598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.758817 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.758913 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.759257 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.759402 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":73}
I20260619 16:46:41.759930 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.738132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.3.233.188:37215)
0619 16:46:41.738238 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.738242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.743191 (+ 4949us) server_negotiation.cc:407] Connection header received
0619 16:46:41.745466 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.745471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.745530 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.745586 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:41.746516 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.747369 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.748684 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.749087 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.749172 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.752637 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:41.752643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:41.752646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:41.752679 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:41.754494 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.755242 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.755246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.755248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.755336 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:41.758866 (+ 3530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:41.758872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:41.758875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:41.759134 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:41.759278 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.759651 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.759782 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260619 16:46:41.760433 11346 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35425
I20260619 16:46:41.760521 11346 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:41.760715 11346 heartbeater.cc:511] Master 127.3.233.190:35425 requested a full tablet report, sending...
W20260619 16:46:41.760921 10106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:41.761332 10107 ts_manager.cc:194] Re-registered known tserver with Master: d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931)
I20260619 16:46:41.762077 10107 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:60603
I20260619 16:46:41.763833 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: NON_VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: true } } }
I20260619 16:46:41.763960 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:41.767073 10856 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: attempting to promote NON_VOTER 5089052ac42544bab6bf1f075fa44d0c to VOTER
I20260619 16:46:41.767432 10856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:41.768702 10902 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 LEARNER]: Refusing update from remote peer 36153ae3d4194954922333389fcf1a0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:46:41.768729 10718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Refusing update from remote peer 36153ae3d4194954922333389fcf1a0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:46:41.769091 11362 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:46:41.769228 11363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:46:41.771188 10856 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.771611 10718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.771742 10902 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.772879 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.772996 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:41.773133 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.773218 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:41.773317 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.773407 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:41.773591 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.773712 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:41.775003 11371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36153ae3d4194954922333389fcf1a0e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:41.775432 11371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:41.787417 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.781593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:33931 (local address 127.0.0.1:48146)
0619 16:46:41.781890 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.781906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.781994 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.782877 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.782884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.782914 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:41.783057 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.783064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.784505 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.784509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.785808 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.785819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.787001 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.787005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.787099 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.787128 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.787178 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.787253 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":5,"thread_start_us":123,"threads_started":1}
I20260619 16:46:41.787930 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.781301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:35289 (local address 127.0.0.1:50112)
0619 16:46:41.781418 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.781434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.781512 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.783722 (+ 2210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.783725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.783752 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:41.783831 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.783836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.785434 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.785437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.786443 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.786450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.787523 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.787528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.787635 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.787670 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.787732 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.787797 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":52,"spinlock_wait_cycles":1920}
I20260619 16:46:41.788255 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.781756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.3.233.131:33931)
0619 16:46:41.782105 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.782118 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.782144 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:46:41.782226 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.782233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.782487 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.782673 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:41.783214 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.784369 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.785951 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.786850 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.787634 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.787751 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.787808 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.787890 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":140,"threads_started":1}
I20260619 16:46:41.788331 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.781511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.3.233.130:35289)
0619 16:46:41.783113 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.783133 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.783158 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:46:41.783244 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.783249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.783512 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.784045 (+ 533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:41.784176 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.785294 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.786563 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.787370 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.787857 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.788016 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.788075 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.788169 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1447,"thread_start_us":153,"threads_started":1}
I20260619 16:46:41.788506 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.782470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39199 (local address 127.0.0.1:53024)
0619 16:46:41.782777 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:41.782807 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:41.782898 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:41.783725 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:41.783728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:41.783756 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:41.783907 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.783917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.785240 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.785243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.786263 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:41.786273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.788105 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.788108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:41.788190 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.788212 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:46:41.788284 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:46:41.788356 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":80,"spinlock_wait_cycles":6016,"thread_start_us":98,"threads_started":1}
I20260619 16:46:41.788895 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:41.782648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.3.233.129:39199)
0619 16:46:41.782997 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:41.783014 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:41.783041 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:46:41.783152 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:41.783156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:41.783396 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:41.783562 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:41.784046 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.785096 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.786401 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:41.787428 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:41.787542 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:41.787709 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:41.788657 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 16:46:41.788761 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260619 16:46:41.791352 11279 tablet_service.cc:1467] Processing CreateTablet for tablet caa4dffbe87a415ab27e5955a709852b (DEFAULT_TABLE table=test-table [id=39ccc9d4250c474c921bb204f7bd4f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:41.791352 11142 tablet_service.cc:1467] Processing CreateTablet for tablet caa4dffbe87a415ab27e5955a709852b (DEFAULT_TABLE table=test-table [id=39ccc9d4250c474c921bb204f7bd4f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:41.791636 11005 tablet_service.cc:1467] Processing CreateTablet for tablet caa4dffbe87a415ab27e5955a709852b (DEFAULT_TABLE table=test-table [id=39ccc9d4250c474c921bb204f7bd4f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:41.792335 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4dffbe87a415ab27e5955a709852b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:41.792397 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4dffbe87a415ab27e5955a709852b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:41.792418 11005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4dffbe87a415ab27e5955a709852b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:41.799620 11377 tablet_bootstrap.cc:492] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Bootstrap starting.
I20260619 16:46:41.800658 11378 tablet_bootstrap.cc:492] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Bootstrap starting.
I20260619 16:46:41.801322 11379 tablet_bootstrap.cc:492] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Bootstrap starting.
I20260619 16:46:41.802904 11377 tablet_bootstrap.cc:654] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:41.802910 11378 tablet_bootstrap.cc:654] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:41.803299 11379 tablet_bootstrap.cc:654] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:41.803855 11377 log.cc:824] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:41.803855 11378 log.cc:824] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:41.803996 11379 log.cc:824] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:41.805992 11379 tablet_bootstrap.cc:492] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: No bootstrap required, opened a new log
I20260619 16:46:41.806000 11378 tablet_bootstrap.cc:492] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: No bootstrap required, opened a new log
I20260619 16:46:41.806000 11377 tablet_bootstrap.cc:492] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: No bootstrap required, opened a new log
I20260619 16:46:41.806209 11379 ts_tablet_manager.cc:1397] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:46:41.806210 11378 ts_tablet_manager.cc:1397] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 16:46:41.806210 11377 ts_tablet_manager.cc:1397] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260619 16:46:41.810354 11379 raft_consensus.cc:348] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:41.810626 11379 raft_consensus.cc:374] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:41.810680 11379 raft_consensus.cc:729] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4831e1ec601748608f6f319cc3402bc9, State: Initialized, Role: FOLLOWER
I20260619 16:46:41.811182 11379 consensus_queue.cc:260] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:41.811139 11378 raft_consensus.cc:348] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:41.811136 11377 raft_consensus.cc:348] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:41.811406 11377 raft_consensus.cc:374] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:41.811406 11378 raft_consensus.cc:374] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:41.811472 11377 raft_consensus.cc:729] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7754ab96594477097a0526a28fbf0f2, State: Initialized, Role: FOLLOWER
I20260619 16:46:41.811465 11378 raft_consensus.cc:729] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c554cdb022914cc284dde53ab49a744b, State: Initialized, Role: FOLLOWER
I20260619 16:46:41.811937 11377 consensus_queue.cc:260] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:41.811937 11378 consensus_queue.cc:260] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:41.812307 11379 ts_tablet_manager.cc:1428] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:46:41.812595 11072 heartbeater.cc:503] Master 127.3.233.190:35425 was elected leader, sending a full tablet report...
I20260619 16:46:41.812945 11378 ts_tablet_manager.cc:1428] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:46:41.812945 11377 ts_tablet_manager.cc:1428] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260619 16:46:41.814121 11209 heartbeater.cc:503] Master 127.3.233.190:35425 was elected leader, sending a full tablet report...
I20260619 16:46:41.815245 11346 heartbeater.cc:503] Master 127.3.233.190:35425 was elected leader, sending a full tablet report...
W20260619 16:46:41.984247 11348 tablet.cc:2369] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:42.014477 11211 tablet.cc:2369] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:42.058308 11074 tablet.cc:2369] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:42.100095 11383 raft_consensus.cc:482] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:42.100346 11383 raft_consensus.cc:504] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:42.101590 11383 leader_election.cc:288] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289), d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931)
I20260619 16:46:42.105716 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.102080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:57747 (local address 127.3.233.131:33931)
0619 16:46:42.102190 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:42.102197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:42.102343 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:46:42.102522 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:42.102525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:42.102586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:42.102662 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:42.102983 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.103485 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.104605 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.105006 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.105135 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.105218 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:42.105500 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:46:42.105567 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:46:42.105754 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.101902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:33829 (local address 127.3.233.130:35289)
0619 16:46:42.102115 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:42.102120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:42.102289 (+ 169us) server_negotiation.cc:407] Connection header received
0619 16:46:42.102466 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:42.102470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:42.102522 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:42.102582 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:42.102905 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.103476 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.104607 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.105004 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.105135 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.105199 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:42.105557 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 16:46:42.105637 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 16:46:42.105773 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.101897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:33931 (local address 127.3.233.129:57747)
0619 16:46:42.102214 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:42.102225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:42.102362 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:42.102699 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:42.102702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:42.102719 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:42.102830 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:42.102835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.103617 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.103620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:42.104481 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:42.104490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.105135 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.105139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:42.105344 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.105381 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:46:42.105448 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:46:42.105560 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":107,"thread_start_us":70,"threads_started":1}
I20260619 16:46:42.105810 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.101795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:35289 (local address 127.3.233.129:33829)
0619 16:46:42.102169 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:42.102184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:42.102341 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:42.102623 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:42.102627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:42.102648 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:42.102775 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:42.102788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.103591 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.103597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:42.104484 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:42.104493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.105135 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.105139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:42.105353 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.105385 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:46:42.105485 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:46:42.105564 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":104,"thread_start_us":94,"threads_started":1}
I20260619 16:46:42.106405 11299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4dffbe87a415ab27e5955a709852b" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7754ab96594477097a0526a28fbf0f2" is_pre_election: true
I20260619 16:46:42.106622 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4dffbe87a415ab27e5955a709852b" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c554cdb022914cc284dde53ab49a744b" is_pre_election: true
I20260619 16:46:42.106802 11299 raft_consensus.cc:2454] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 0.
I20260619 16:46:42.106962 11162 raft_consensus.cc:2454] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 0.
I20260619 16:46:42.107374 10960 leader_election.cc:302] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4831e1ec601748608f6f319cc3402bc9, d7754ab96594477097a0526a28fbf0f2; no voters:
I20260619 16:46:42.107712 11383 raft_consensus.cc:2781] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:42.107797 11383 raft_consensus.cc:482] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:42.107841 11383 raft_consensus.cc:3037] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:42.108984 11383 raft_consensus.cc:504] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:42.109324 11383 leader_election.cc:288] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 election: Requested vote from peers c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289), d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931)
I20260619 16:46:42.109730 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4dffbe87a415ab27e5955a709852b" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c554cdb022914cc284dde53ab49a744b"
I20260619 16:46:42.109728 11299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4dffbe87a415ab27e5955a709852b" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7754ab96594477097a0526a28fbf0f2"
I20260619 16:46:42.109871 11162 raft_consensus.cc:3037] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:42.109879 11299 raft_consensus.cc:3037] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:42.110992 11162 raft_consensus.cc:2454] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 1.
I20260619 16:46:42.111089 11299 raft_consensus.cc:2454] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 1.
I20260619 16:46:42.111299 10959 leader_election.cc:302] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4831e1ec601748608f6f319cc3402bc9, c554cdb022914cc284dde53ab49a744b; no voters:
I20260619 16:46:42.111500 11383 raft_consensus.cc:2781] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:42.111708 11383 raft_consensus.cc:686] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 4831e1ec601748608f6f319cc3402bc9, State: Running, Role: LEADER
I20260619 16:46:42.112102 11383 consensus_queue.cc:237] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } }
I20260619 16:46:42.115685 10106 catalog_manager.cc:5495] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4831e1ec601748608f6f319cc3402bc9 (127.3.233.129). New cstate: current_term: 1 leader_uuid: "4831e1ec601748608f6f319cc3402bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } health_report { overall_health: HEALTHY } } }
I20260619 16:46:42.152396 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.148417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.3.233.129:39199)
0619 16:46:42.148566 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:42.148571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:42.148586 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:42.148715 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:42.148718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:42.148773 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:42.148893 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:42.149147 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.149719 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.150753 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.151007 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.151123 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.151442 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:42.151894 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:42.152055 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:42.152092 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 16:46:42.152156 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:46:42.158617 11299 raft_consensus.cc:1261] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Refusing update from remote peer 4831e1ec601748608f6f319cc3402bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:42.158684 11162 raft_consensus.cc:1261] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Refusing update from remote peer 4831e1ec601748608f6f319cc3402bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:42.159488 11388 consensus_queue.cc:1035] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:42.159751 11383 consensus_queue.cc:1035] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:42.172649 11393 mvcc.cc:205] Tried to move back new op lower bound from 7298611618434326528 to 7298611618253709312. Current Snapshot: MvccSnapshot[applied={T|T < 7298611618434326528}]
I20260619 16:46:42.178877 10106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:42.179162 10106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:42.183985 10106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36153ae3d4194954922333389fcf1a0e: Sending DeleteTablet for 3 replicas of tablet caa4dffbe87a415ab27e5955a709852b
I20260619 16:46:42.184898 11142 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4dffbe87a415ab27e5955a709852b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:42 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50112
I20260619 16:46:42.185112 11005 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4dffbe87a415ab27e5955a709852b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:42 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:53024
I20260619 16:46:42.185719 11279 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4dffbe87a415ab27e5955a709852b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:42 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:48146
I20260619 16:46:42.186205 11405 tablet_replica.cc:330] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: stopping tablet replica
I20260619 16:46:42.186589 11405 raft_consensus.cc:2229] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:42.186983 11405 raft_consensus.cc:2258] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:42.188580 11405 ts_tablet_manager.cc:1905] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:42.188982 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 10075
I20260619 16:46:42.191704 11405 ts_tablet_manager.cc:1918] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:42.191820 11405 log.cc:1196] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/caa4dffbe87a415ab27e5955a709852b
I20260619 16:46:42.192119 11405 ts_tablet_manager.cc:1939] T caa4dffbe87a415ab27e5955a709852b P d7754ab96594477097a0526a28fbf0f2: Deleting consensus metadata
I20260619 16:46:42.203157 11402 tablet_replica.cc:330] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: stopping tablet replica
I20260619 16:46:42.203599 11402 raft_consensus.cc:2229] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:42.203965 11402 raft_consensus.cc:2258] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:42.204391 11403 tablet_replica.cc:330] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: stopping tablet replica
I20260619 16:46:42.204751 11403 raft_consensus.cc:2229] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:42.205166 11403 raft_consensus.cc:2258] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:42.206806 11402 ts_tablet_manager.cc:1905] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:42.206799 11403 ts_tablet_manager.cc:1905] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:42.207903 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.188024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.3.233.189:43481)
0619 16:46:42.188139 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:42.188143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:42.188381 (+ 238us) server_negotiation.cc:407] Connection header received
0619 16:46:42.188572 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:42.188576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:42.188626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:42.188677 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:42.199443 (+ 10766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.200273 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.201505 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.201878 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.201996 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.203762 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:42.203766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:42.203769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:42.203797 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:42.205690 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.206253 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.206256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.206258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.206349 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.206685 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.206688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.206690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.206959 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:42.207075 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:42.207606 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 16:46:42.207745 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:46:42.209707 11403 ts_tablet_manager.cc:1918] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:42.209731 11402 ts_tablet_manager.cc:1918] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:42.209820 11403 log.cc:1196] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/caa4dffbe87a415ab27e5955a709852b
I20260619 16:46:42.209829 11402 log.cc:1196] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/caa4dffbe87a415ab27e5955a709852b
I20260619 16:46:42.210165 11403 ts_tablet_manager.cc:1939] T caa4dffbe87a415ab27e5955a709852b P 4831e1ec601748608f6f319cc3402bc9: Deleting consensus metadata
I20260619 16:46:42.210170 11402 ts_tablet_manager.cc:1939] T caa4dffbe87a415ab27e5955a709852b P c554cdb022914cc284dde53ab49a744b: Deleting consensus metadata
W20260619 16:46:42.211277 10960 connection.cc:441] server connection from 127.0.0.1:53024 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53024 (request call id 1) could send its response
W20260619 16:46:42.211378 11098 connection.cc:441] server connection from 127.0.0.1:50112 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50112 (request call id 1) could send its response
I20260619 16:46:42.220661 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.187574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.3.233.188:37215)
0619 16:46:42.187701 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:42.187706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:42.209740 (+ 22034us) server_negotiation.cc:407] Connection header received
0619 16:46:42.210124 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:42.210130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:42.210188 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:42.210247 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:42.211221 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.211971 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.213418 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.213782 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.213882 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.215972 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:42.215977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:42.215979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:42.216008 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:42.218096 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.218864 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.218867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.218869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.218988 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.219281 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.219285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.219287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.219485 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:42.219595 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:42.219914 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:46:42.220471 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:46:42.222355 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.212717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.3.233.188:37215)
0619 16:46:42.212846 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:42.212850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:42.213083 (+ 233us) server_negotiation.cc:407] Connection header received
0619 16:46:42.213247 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:42.213250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:42.213293 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:42.213341 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:42.214194 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.215018 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.216172 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.216509 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.216601 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.218480 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:42.218485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:42.218490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:42.218517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:42.220589 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.221077 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.221081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.221082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.221159 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.221445 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.221449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.221451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.221634 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:42.221737 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:42.222102 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 16:46:42.222211 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:46:42.223433 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:42.212377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.3.233.189:43481)
0619 16:46:42.212518 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:42.212523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:42.212956 (+ 433us) server_negotiation.cc:407] Connection header received
0619 16:46:42.213107 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:42.213110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:42.213150 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:42.213197 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:42.214776 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.215664 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.217235 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:42.217637 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:42.217721 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:42.219845 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:42.219849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:42.219852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:42.219879 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:42.221565 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.222014 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.222021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.222023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.222109 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:42.222420 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:42.222434 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:42.222437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:42.222659 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:42.222756 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:42.223161 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 16:46:42.223284 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:46:43.171881 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.171274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:44549)
0619 16:46:43.171650 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.171762 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":73,"threads_started":1}
W20260619 16:46:43.172724 11072 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:35425 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.174193 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.173450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:56249)
0619 16:46:43.173951 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.174049 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":66,"threads_started":1}
W20260619 16:46:43.174619 11346 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:35425 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.176424 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.175482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:35651)
0619 16:46:43.176230 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.176316 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":67,"threads_started":1}
W20260619 16:46:43.176952 11209 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:35425 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.684659 11415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36153ae3d4194954922333389fcf1a0e)
I20260619 16:46:43.684703 11416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36153ae3d4194954922333389fcf1a0e)
I20260619 16:46:43.684857 11415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:43.684913 11416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:43.685438 11415 leader_election.cc:288] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425), 594f84c57e764815bafef69f8e073aa3 (127.3.233.189:43481)
I20260619 16:46:43.686513 11416 leader_election.cc:288] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425), 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188:37215)
I20260619 16:46:43.686563 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.685690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:44890)
0619 16:46:43.686352 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.686443 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":67,"threads_started":1}
I20260619 16:46:43.686882 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.686190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:44894)
0619 16:46:43.686718 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.686794 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":77,"threads_started":1}
W20260619 16:46:43.686892 10878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
W20260619 16:46:43.687484 10694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.691310 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.686219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:37215 (local address 127.0.0.1:40656)
0619 16:46:43.687067 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.687079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:43.687163 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:43.687622 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:43.687625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:43.687642 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:43.687719 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:43.687729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.688755 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.688758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:43.690021 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:43.690032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.690894 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.690897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:43.691018 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:43.691037 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:46:43.691089 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 16:46:43.691169 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":3,"thread_start_us":50,"threads_started":1}
I20260619 16:46:43.691354 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.685780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:43481 (local address 127.0.0.1:37176)
0619 16:46:43.686745 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.686759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:43.686865 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:43.687523 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:43.687527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:43.687547 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:43.687696 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:43.687704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.688755 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.688758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:43.690021 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:43.690032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.690997 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.691001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:43.691092 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:43.691115 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:46:43.691171 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 16:46:43.691234 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":67,"threads_started":1}
I20260619 16:46:43.691892 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.686575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.3.233.188:37215)
0619 16:46:43.687395 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:43.687402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:43.687415 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:43.687453 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:43.687455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:43.687514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:43.687820 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:43.687896 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.688640 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.690171 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.690775 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.691629 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:43.691680 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:43.691714 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 16:46:43.691777 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":64,"threads_started":1}
I20260619 16:46:43.692217 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.686084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.3.233.189:43481)
0619 16:46:43.687232 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:43.687239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:43.687253 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:43.687297 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:43.687300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:43.687341 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:43.687415 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:43.687813 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.688626 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.690171 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:43.690764 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:43.690875 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:43.691897 (+ 1022us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:43.692033 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:46:43.692111 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":940,"server-negotiator.queue_time_us":1081,"thread_start_us":67,"threads_started":1}
I20260619 16:46:43.692242 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "594f84c57e764815bafef69f8e073aa3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5089052ac42544bab6bf1f075fa44d0c" is_pre_election: true
I20260619 16:46:43.692452 10902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 594f84c57e764815bafef69f8e073aa3 in term 1.
I20260619 16:46:43.692655 10718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5089052ac42544bab6bf1f075fa44d0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "594f84c57e764815bafef69f8e073aa3" is_pre_election: true
I20260619 16:46:43.692783 10718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5089052ac42544bab6bf1f075fa44d0c in term 1.
I20260619 16:46:43.692854 10695 leader_election.cc:302] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5089052ac42544bab6bf1f075fa44d0c, 594f84c57e764815bafef69f8e073aa3; no voters:
I20260619 16:46:43.692894 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.692436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:44910)
0619 16:46:43.692748 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.692816 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
W20260619 16:46:43.692999 10878 leader_election.cc:334] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425): Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.693117 10879 leader_election.cc:302] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5089052ac42544bab6bf1f075fa44d0c, 594f84c57e764815bafef69f8e073aa3; no voters: 36153ae3d4194954922333389fcf1a0e
I20260619 16:46:43.693303 11415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:46:43.693318 11416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:46:43.693362 11415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36153ae3d4194954922333389fcf1a0e)
I20260619 16:46:43.693367 11416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36153ae3d4194954922333389fcf1a0e)
I20260619 16:46:43.693409 11416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:46:43.693409 11415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:46:43.694628 11415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:43.694628 11416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:43.695003 11416 leader_election.cc:288] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [CANDIDATE]: Term 2 election: Requested vote from peers 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425), 5089052ac42544bab6bf1f075fa44d0c (127.3.233.188:37215)
I20260619 16:46:43.695015 11415 leader_election.cc:288] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 2 election: Requested vote from peers 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425), 594f84c57e764815bafef69f8e073aa3 (127.3.233.189:43481)
I20260619 16:46:43.695454 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.693420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:44912)
0619 16:46:43.695326 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.695382 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1820}
I20260619 16:46:43.695575 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.695222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:44924)
0619 16:46:43.695422 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.695488 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 16:46:43.695621 10718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5089052ac42544bab6bf1f075fa44d0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "594f84c57e764815bafef69f8e073aa3"
W20260619 16:46:43.695662 10694 leader_election.cc:334] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425): Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.695755 10718 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5089052ac42544bab6bf1f075fa44d0c in current term 2: Already voted for candidate 594f84c57e764815bafef69f8e073aa3 in this term.
I20260619 16:46:43.696017 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "594f84c57e764815bafef69f8e073aa3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5089052ac42544bab6bf1f075fa44d0c"
I20260619 16:46:43.696172 10902 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 594f84c57e764815bafef69f8e073aa3 in current term 2: Already voted for candidate 5089052ac42544bab6bf1f075fa44d0c in this term.
I20260619 16:46:43.696450 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.696103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:44934)
0619 16:46:43.696339 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.696393 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260619 16:46:43.696588 10694 leader_election.cc:334] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425): Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.696645 10694 leader_election.cc:302] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 594f84c57e764815bafef69f8e073aa3; no voters: 36153ae3d4194954922333389fcf1a0e, 5089052ac42544bab6bf1f075fa44d0c
I20260619 16:46:43.696735 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:43.696521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:44936)
0619 16:46:43.696629 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:43.696678 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 16:46:43.696802 10878 leader_election.cc:334] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425): Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:43.696799 11416 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:46:43.696849 10878 leader_election.cc:302] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5089052ac42544bab6bf1f075fa44d0c; no voters: 36153ae3d4194954922333389fcf1a0e, 594f84c57e764815bafef69f8e073aa3
I20260619 16:46:43.697053 11415 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:46:44.174038 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:44.173482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:50603)
0619 16:46:44.173795 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:44.173914 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":78,"threads_started":1}
I20260619 16:46:44.176029 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:44.175372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:47287)
0619 16:46:44.175794 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:44.175886 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":80,"threads_started":1}
I20260619 16:46:44.177929 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:44.177537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:46235)
0619 16:46:44.177777 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:44.177839 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
I20260619 16:46:45.175894 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.175199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:55561)
0619 16:46:45.175494 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.175747 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260619 16:46:45.177188 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.176718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:46243)
0619 16:46:45.177018 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.177086 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":62,"threads_started":1}
I20260619 16:46:45.179075 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.178556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:50589)
0619 16:46:45.178813 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.178994 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":114,"threads_started":1}
I20260619 16:46:45.312551 11431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:45.312685 11431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:45.313202 11431 leader_election.cc:288] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425), 594f84c57e764815bafef69f8e073aa3 (127.3.233.189:43481)
I20260619 16:46:45.313670 10718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5089052ac42544bab6bf1f075fa44d0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "594f84c57e764815bafef69f8e073aa3" is_pre_election: true
I20260619 16:46:45.313835 10718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5089052ac42544bab6bf1f075fa44d0c in term 2.
I20260619 16:46:45.313869 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.313448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:45142)
0619 16:46:45.313691 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.313751 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260619 16:46:45.314203 10879 leader_election.cc:302] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5089052ac42544bab6bf1f075fa44d0c, 594f84c57e764815bafef69f8e073aa3; no voters:
I20260619 16:46:45.314396 11431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 16:46:45.314476 11431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:45.314534 11431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:46:45.314772 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.314528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:45150)
0619 16:46:45.314632 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.314695 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 16:46:45.314972 10878 leader_election.cc:334] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425): Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:45.315738 11431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:45.316099 11431 leader_election.cc:288] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 3 election: Requested vote from peers 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425), 594f84c57e764815bafef69f8e073aa3 (127.3.233.189:43481)
I20260619 16:46:45.316416 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.316214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:45160)
0619 16:46:45.316300 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.316361 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 16:46:45.316449 10718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5089052ac42544bab6bf1f075fa44d0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "594f84c57e764815bafef69f8e073aa3"
I20260619 16:46:45.316601 10718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:46:45.317113 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.316912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:45172)
0619 16:46:45.317017 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.317069 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:46:45.317210 10878 leader_election.cc:334] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425): Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
I20260619 16:46:45.317629 10718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5089052ac42544bab6bf1f075fa44d0c in term 3.
I20260619 16:46:45.317939 10879 leader_election.cc:302] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5089052ac42544bab6bf1f075fa44d0c, 594f84c57e764815bafef69f8e073aa3; no voters: 36153ae3d4194954922333389fcf1a0e
I20260619 16:46:45.318156 11431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:46:45.318372 11431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [term 3 LEADER]: Becoming Leader. State: Replica: 5089052ac42544bab6bf1f075fa44d0c, State: Running, Role: LEADER
I20260619 16:46:45.318662 11431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } }
I20260619 16:46:45.320021 11434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5089052ac42544bab6bf1f075fa44d0c. Latest consensus state: current_term: 3 leader_uuid: "5089052ac42544bab6bf1f075fa44d0c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:45.320153 11434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:45.320510 11436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:45.322695 11436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:45.323167 11436 catalog_manager.cc:1259] Loaded cluster ID: 0022aa6f49db4288b0e5d64c7990ba81
I20260619 16:46:45.323221 11436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:45.323851 11436 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:45.324275 11436 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Loaded TSK: 0
I20260619 16:46:45.324951 11436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:46:45.368034 10892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:45.371713 10892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:45.374886 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.374601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:45186)
0619 16:46:45.374727 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.374794 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 16:46:45.375293 10718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [term 3 FOLLOWER]: Refusing update from remote peer 5089052ac42544bab6bf1f075fa44d0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 16:46:45.375648 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.375414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.0.0.1:45202)
0619 16:46:45.375540 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.375595 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:46:45.375768 11434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 16:46:45.375942 10878 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c -> Peer 36153ae3d4194954922333389fcf1a0e (127.3.233.190:35425): Couldn't send request to peer 36153ae3d4194954922333389fcf1a0e. Status: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:46:45.377101 11438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5089052ac42544bab6bf1f075fa44d0c. Latest consensus state: current_term: 3 leader_uuid: "5089052ac42544bab6bf1f075fa44d0c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:45.377238 11438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:45.378340 11434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5089052ac42544bab6bf1f075fa44d0c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:45.378499 11434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:45.378661 11438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5089052ac42544bab6bf1f075fa44d0c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36153ae3d4194954922333389fcf1a0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35425 } } peers { permanent_uuid: "594f84c57e764815bafef69f8e073aa3" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 43481 } attrs { promote: false } } peers { permanent_uuid: "5089052ac42544bab6bf1f075fa44d0c" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 37215 } attrs { promote: false } } }
I20260619 16:46:45.378824 11438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594f84c57e764815bafef69f8e073aa3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:46:45.379138 11437 mvcc.cc:205] Tried to move back new op lower bound from 7298611631610138624 to 7298611631387803648. Current Snapshot: MvccSnapshot[applied={T|T < 7298611631610138624 or (T in {7298611631610138624})}]
I20260619 16:46:45.379305 11439 mvcc.cc:205] Tried to move back new op lower bound from 7298611631610138624 to 7298611631387803648. Current Snapshot: MvccSnapshot[applied={T|T < 7298611631610138624 or (T in {7298611631610138624})}]
I20260619 16:46:45.389155 11443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.384673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.3.233.131:33931)
0619 16:46:45.385128 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:45.385134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:45.385148 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:45.385196 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:45.385199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:45.385272 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:45.385385 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:45.385620 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.386391 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.387874 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.388487 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.388583 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.388658 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:45.388962 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 16:46:45.389040 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":95,"threads_started":1}
I20260619 16:46:45.389155 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.384544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:33931 (local address 127.0.0.1:34612)
0619 16:46:45.384811 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.384827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:45.384918 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:45.385397 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:45.385400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:45.385415 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:45.385523 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.385529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.386492 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.386497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.387760 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.387770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.388694 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.388698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.388846 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.388875 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:46:45.388948 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:46:45.389017 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":22}
I20260619 16:46:45.389243 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.385521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.3.233.129:39199)
0619 16:46:45.385755 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:45.385770 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:45.385784 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:45.385834 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:45.385836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:45.385894 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:45.385993 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:45.386229 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.386744 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.388062 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.388659 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.388772 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.388842 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:45.389043 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 16:46:45.389111 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260619 16:46:45.389662 11444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.384781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39199 (local address 127.0.0.1:51378)
0619 16:46:45.385002 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.385015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:45.385087 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:45.386014 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:45.386018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:45.386036 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:45.386124 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.386129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.387108 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.387110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.387830 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.387836 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.388772 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.388775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.388909 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.388936 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:46:45.389028 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:46:45.389098 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":21,"thread_start_us":91,"threads_started":1}
I20260619 16:46:45.389787 11279 tablet_service.cc:1467] Processing CreateTablet for tablet 7a866442753c4e0f8811951963c69691 (DEFAULT_TABLE table=test-table [id=67e3c525a0b04bf3908c5f95ad8e637c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:45.390075 11279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a866442753c4e0f8811951963c69691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:45.390210 11005 tablet_service.cc:1467] Processing CreateTablet for tablet 7a866442753c4e0f8811951963c69691 (DEFAULT_TABLE table=test-table [id=67e3c525a0b04bf3908c5f95ad8e637c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:45.390489 11005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a866442753c4e0f8811951963c69691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:45.391837 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.385292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:35289 (local address 127.0.0.1:41908)
0619 16:46:45.385842 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.385854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:45.385933 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:45.386852 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:45.386858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:45.386878 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:45.387008 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.387013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.388816 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.388819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.390514 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:45.390524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.391417 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.391421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:45.391506 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.391533 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:46:45.391621 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:46:45.391686 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":75,"threads_started":1}
I20260619 16:46:45.391860 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.385399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.3.233.130:35289)
0619 16:46:45.386621 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:45.386626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:45.386640 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:45.386684 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:45.386687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:45.386731 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:45.386815 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:45.387125 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.388702 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.390657 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.391290 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.391405 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.391465 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:45.391649 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:46:45.391734 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260619 16:46:45.392630 11142 tablet_service.cc:1467] Processing CreateTablet for tablet 7a866442753c4e0f8811951963c69691 (DEFAULT_TABLE table=test-table [id=67e3c525a0b04bf3908c5f95ad8e637c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:45.392889 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a866442753c4e0f8811951963c69691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:45.392997 11448 tablet_bootstrap.cc:492] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: Bootstrap starting.
I20260619 16:46:45.393092 11449 tablet_bootstrap.cc:492] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9: Bootstrap starting.
I20260619 16:46:45.393849 11448 tablet_bootstrap.cc:654] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:45.393894 11449 tablet_bootstrap.cc:654] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:45.395109 11448 tablet_bootstrap.cc:492] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: No bootstrap required, opened a new log
I20260619 16:46:45.395193 11448 ts_tablet_manager.cc:1397] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:45.395186 11452 tablet_bootstrap.cc:492] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: Bootstrap starting.
I20260619 16:46:45.395598 11448 raft_consensus.cc:348] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.395691 11448 raft_consensus.cc:374] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:45.395712 11448 raft_consensus.cc:729] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7754ab96594477097a0526a28fbf0f2, State: Initialized, Role: FOLLOWER
I20260619 16:46:45.395848 11448 consensus_queue.cc:260] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.396080 11452 tablet_bootstrap.cc:654] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:45.396242 11448 ts_tablet_manager.cc:1428] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:45.396826 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.396538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:33985)
0619 16:46:45.396640 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.396698 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 16:46:45.397233 11449 tablet_bootstrap.cc:492] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9: No bootstrap required, opened a new log
I20260619 16:46:45.397317 11449 ts_tablet_manager.cc:1397] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 16:46:45.397719 11449 raft_consensus.cc:348] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.397833 11449 raft_consensus.cc:374] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:45.397868 11449 raft_consensus.cc:729] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4831e1ec601748608f6f319cc3402bc9, State: Initialized, Role: FOLLOWER
I20260619 16:46:45.398007 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.397263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:43153)
0619 16:46:45.397888 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.397948 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551}
I20260619 16:46:45.398002 11449 consensus_queue.cc:260] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.398379 11449 ts_tablet_manager.cc:1428] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:46:45.398999 11345 heartbeater.cc:503] Master 127.3.233.188:37215 was elected leader, sending a full tablet report...
I20260619 16:46:45.399423 11452 tablet_bootstrap.cc:492] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: No bootstrap required, opened a new log
I20260619 16:46:45.399549 11452 ts_tablet_manager.cc:1397] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 16:46:45.399904 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.399688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:46199)
0619 16:46:45.399787 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.399838 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 16:46:45.400029 11452 raft_consensus.cc:348] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.400135 11452 raft_consensus.cc:374] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:45.400177 11452 raft_consensus.cc:729] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c554cdb022914cc284dde53ab49a744b, State: Initialized, Role: FOLLOWER
I20260619 16:46:45.400300 11452 consensus_queue.cc:260] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.400647 11071 heartbeater.cc:503] Master 127.3.233.188:37215 was elected leader, sending a full tablet report...
I20260619 16:46:45.400679 11452 ts_tablet_manager.cc:1428] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:46:45.401602 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.401073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:37867)
0619 16:46:45.401472 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.401535 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
I20260619 16:46:45.401710 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.400834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:60011)
0619 16:46:45.401579 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.401634 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679}
I20260619 16:46:45.402328 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.402083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:33683)
0619 16:46:45.402219 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.402266 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 16:46:45.545696 11455 raft_consensus.cc:482] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:45.545884 11455 raft_consensus.cc:504] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.546449 11455 leader_election.cc:288] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931), c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289)
I20260619 16:46:45.546984 11299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a866442753c4e0f8811951963c69691" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7754ab96594477097a0526a28fbf0f2" is_pre_election: true
I20260619 16:46:45.547091 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a866442753c4e0f8811951963c69691" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c554cdb022914cc284dde53ab49a744b" is_pre_election: true
I20260619 16:46:45.547163 11299 raft_consensus.cc:2454] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 0.
I20260619 16:46:45.547268 11162 raft_consensus.cc:2454] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 0.
I20260619 16:46:45.547580 10960 leader_election.cc:302] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4831e1ec601748608f6f319cc3402bc9, d7754ab96594477097a0526a28fbf0f2; no voters:
I20260619 16:46:45.547761 11455 raft_consensus.cc:2781] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:45.547824 11455 raft_consensus.cc:482] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:45.547866 11455 raft_consensus.cc:3037] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:45.548971 11455 raft_consensus.cc:504] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.549348 11455 leader_election.cc:288] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 election: Requested vote from peers d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931), c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289)
I20260619 16:46:45.549700 11299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a866442753c4e0f8811951963c69691" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7754ab96594477097a0526a28fbf0f2"
I20260619 16:46:45.549841 11299 raft_consensus.cc:3037] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:45.549788 11162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a866442753c4e0f8811951963c69691" candidate_uuid: "4831e1ec601748608f6f319cc3402bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c554cdb022914cc284dde53ab49a744b"
I20260619 16:46:45.549932 11162 raft_consensus.cc:3037] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:45.550808 11299 raft_consensus.cc:2454] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 1.
I20260619 16:46:45.550884 11162 raft_consensus.cc:2454] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4831e1ec601748608f6f319cc3402bc9 in term 1.
I20260619 16:46:45.551189 10960 leader_election.cc:302] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4831e1ec601748608f6f319cc3402bc9, d7754ab96594477097a0526a28fbf0f2; no voters:
I20260619 16:46:45.551373 11455 raft_consensus.cc:2781] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:45.551646 11455 raft_consensus.cc:686] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 4831e1ec601748608f6f319cc3402bc9, State: Running, Role: LEADER
I20260619 16:46:45.552042 11455 consensus_queue.cc:237] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } }
I20260619 16:46:45.552461 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.552141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:58379)
0619 16:46:45.552306 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.552376 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 16:46:45.554033 10892 catalog_manager.cc:5495] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4831e1ec601748608f6f319cc3402bc9 (127.3.233.129). New cstate: current_term: 1 leader_uuid: "4831e1ec601748608f6f319cc3402bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4831e1ec601748608f6f319cc3402bc9" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:45.582036 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.579061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.3.233.129:39199)
0619 16:46:45.579174 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:45.579178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:45.579192 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:45.579305 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:45.579308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:45.579352 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:45.579430 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:45.579731 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.580238 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.581103 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:45.581313 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:45.581387 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:45.581619 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:45.581684 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:45.581734 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:45.581813 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 16:46:45.581857 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:46:45.584558 11162 raft_consensus.cc:1261] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Refusing update from remote peer 4831e1ec601748608f6f319cc3402bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:45.584537 11299 raft_consensus.cc:1261] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Refusing update from remote peer 4831e1ec601748608f6f319cc3402bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:45.585052 11455 consensus_queue.cc:1035] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7754ab96594477097a0526a28fbf0f2" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:45.585242 11457 consensus_queue.cc:1035] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c554cdb022914cc284dde53ab49a744b" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 35289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:45.586221 11208 heartbeater.cc:503] Master 127.3.233.188:37215 was elected leader, sending a full tablet report...
I20260619 16:46:45.587028 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.586698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:57685)
0619 16:46:45.586858 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.586951 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 16:46:45.587553 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.587274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:47185)
0619 16:46:45.587430 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.587485 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 16:46:45.590102 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.589851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:50795)
0619 16:46:45.589971 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.590029 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 16:46:45.591270 11430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.590935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.130:60363)
0619 16:46:45.591143 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.591197 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 16:46:45.591467 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.591176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.131:50995)
0619 16:46:45.591355 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.591402 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 16:46:45.592902 11462 mvcc.cc:205] Tried to move back new op lower bound from 7298611632469151744 to 7298611632343068672. Current Snapshot: MvccSnapshot[applied={T|T < 7298611632469151744}]
I20260619 16:46:45.594830 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.594558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35425 (local address 127.3.233.129:58045)
0619 16:46:45.594704 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.594757 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 16:46:45.594830 10891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:45.594993 10891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:45.598841 10891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5089052ac42544bab6bf1f075fa44d0c: Sending DeleteTablet for 3 replicas of tablet 7a866442753c4e0f8811951963c69691
I20260619 16:46:45.599480 11279 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a866442753c4e0f8811951963c69691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:45 UTC) from {username='oryx', principal='oryx/127.3.233.188@KRBTEST.COM'} at 127.0.0.1:34612
I20260619 16:46:45.599655 11142 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a866442753c4e0f8811951963c69691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:45 UTC) from {username='oryx', principal='oryx/127.3.233.188@KRBTEST.COM'} at 127.0.0.1:41908
I20260619 16:46:45.599829 11005 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a866442753c4e0f8811951963c69691 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:45 UTC) from {username='oryx', principal='oryx/127.3.233.188@KRBTEST.COM'} at 127.0.0.1:51378
I20260619 16:46:45.600044 11471 tablet_replica.cc:330] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: stopping tablet replica
I20260619 16:46:45.600174 11471 raft_consensus.cc:2229] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:45.600183 11472 tablet_replica.cc:330] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9: stopping tablet replica
I20260619 16:46:45.600178 11470 tablet_replica.cc:330] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: stopping tablet replica
I20260619 16:46:45.600317 11471 raft_consensus.cc:2258] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:45.600333 11472 raft_consensus.cc:2229] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:45.600373 11470 raft_consensus.cc:2229] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:45.600605 11472 raft_consensus.cc:2258] T 7a866442753c4e0f8811951963c69691 P 4831e1ec601748608f6f319cc3402bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:45.600750 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 10852
I20260619 16:46:45.600874 11470 raft_consensus.cc:2258] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:45.601527 11471 ts_tablet_manager.cc:1905] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:45.602802 11470 ts_tablet_manager.cc:1905] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:45.603848 11471 ts_tablet_manager.cc:1918] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:45.603938 11471 log.cc:1196] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/7a866442753c4e0f8811951963c69691
I20260619 16:46:45.604182 11471 ts_tablet_manager.cc:1939] T 7a866442753c4e0f8811951963c69691 P c554cdb022914cc284dde53ab49a744b: Deleting consensus metadata
I20260619 16:46:45.604874 10877 catalog_manager.cc:4841] TS c554cdb022914cc284dde53ab49a744b (127.3.233.130:35289): tablet 7a866442753c4e0f8811951963c69691 (table test-table [id=67e3c525a0b04bf3908c5f95ad8e637c]) successfully deleted
I20260619 16:46:45.605878 11470 ts_tablet_manager.cc:1918] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:45.605974 11470 log.cc:1196] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/7a866442753c4e0f8811951963c69691
I20260619 16:46:45.606253 11470 ts_tablet_manager.cc:1939] T 7a866442753c4e0f8811951963c69691 P d7754ab96594477097a0526a28fbf0f2: Deleting consensus metadata
I20260619 16:46:45.607178 10878 catalog_manager.cc:4841] TS d7754ab96594477097a0526a28fbf0f2 (127.3.233.131:33931): tablet 7a866442753c4e0f8811951963c69691 (table test-table [id=67e3c525a0b04bf3908c5f95ad8e637c]) successfully deleted
I20260619 16:46:45.610690 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 11079
I20260619 16:46:45.610765 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:45.610476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39199 (local address 127.0.0.1:51406)
0619 16:46:45.610638 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:45.610690 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:46:45.611348 10876 catalog_manager.cc:4584] TS 4831e1ec601748608f6f319cc3402bc9 (127.3.233.129:39199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a866442753c4e0f8811951963c69691: Network error: Client connection negotiation failed: client connection to 127.3.233.129:39199: connect: Connection refused (error 111)
I20260619 16:46:45.617638 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 11216
I20260619 16:46:45.624584 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 10571
I20260619 16:46:45.631981 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 10680
2026-06-19T16:46:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7361 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:45 dist-test-slave-cgwx krb5kdc[11475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:45 dist-test-slave-cgwx krb5kdc[11475](info): set up 2 sockets
Jun 19 16:46:45 dist-test-slave-cgwx krb5kdc[11475](info): commencing operation
krb5kdc: starting...
W20260619 16:46:46.716017 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:46 dist-test-slave-cgwx krb5kdc[11475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab.
Entry for principal oryx/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab.
I20260619 16:46:46.842182 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:43439
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab
--principal=oryx/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:46.955068 11488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:46.955369 11488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:46.955456 11488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:46.961124 11488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:46.961211 11488 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:46:46.961236 11488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:46.961261 11488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:46.961292 11488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:46.961323 11488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:46.967242 11488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:43439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.190.keytab
--principal=oryx/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:43439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:46.968292 11488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:46.969312 11488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:46.977921 11494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:46.977970 11493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:46.977965 11496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:46.978228 11488 server_base.cc:1034] running on GCE node
I20260619 16:46:46.978833 11488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:46.980062 11488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:46.981256 11488 hybrid_clock.cc:648] HybridClock initialized: now 1781887606981243 us; error 37 us; skew 500 ppm
Jun 19 16:46:46 dist-test-slave-cgwx krb5kdc[11475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:46.984198 11488 init.cc:377] Logged in from keytab as oryx/127.3.233.190@KRBTEST.COM (short username oryx)
I20260619 16:46:46.985033 11488 webserver.cc:460] Webserver started at http://127.3.233.190:41853/ using document root <none> and password file <none>
I20260619 16:46:46.985556 11488 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:46.985612 11488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:46.985780 11488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:46.987818 11488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "20efe0fd88104e47903095aed754ebc4"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-cgwx"
server_key: "4b659513b845dcae4c4c8fc9ad55391d"
server_key_iv: "9360a7b8f0abda99339875d171e7749f"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.988315 11488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "20efe0fd88104e47903095aed754ebc4"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-cgwx"
server_key: "4b659513b845dcae4c4c8fc9ad55391d"
server_key_iv: "9360a7b8f0abda99339875d171e7749f"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.992043 11488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:46:46.994431 11503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:46.995852 11488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:46.995995 11488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "20efe0fd88104e47903095aed754ebc4"
format_stamp: "Formatted at 2026-06-19 16:46:46 on dist-test-slave-cgwx"
server_key: "4b659513b845dcae4c4c8fc9ad55391d"
server_key_iv: "9360a7b8f0abda99339875d171e7749f"
server_key_version: "encryptionkey@0"
I20260619 16:46:46.996121 11488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:47.009840 11488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:47.013485 11488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:47.013715 11488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:47.022095 11488 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:43439
I20260619 16:46:47.022116 11565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:43439 every 8 connection(s)
I20260619 16:46:47.023223 11488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:47.026075 11566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:47.028877 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 11488
I20260619 16:46:47.029057 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:47.029349 4006 external_mini_cluster.cc:1428] Setting key 614fbf39926ff6846666a5e3877f1337
I20260619 16:46:47.032080 11566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Bootstrap starting.
I20260619 16:46:47.034265 11566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:47.035050 11566 log.cc:824] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:47.036852 11566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: No bootstrap required, opened a new log
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:47.040585 11566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20efe0fd88104e47903095aed754ebc4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43439 } }
I20260619 16:46:47.040835 11566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:47.040918 11566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20efe0fd88104e47903095aed754ebc4, State: Initialized, Role: FOLLOWER
I20260619 16:46:47.041409 11566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20efe0fd88104e47903095aed754ebc4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43439 } }
I20260619 16:46:47.041563 11566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:47.041651 11566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:47.041751 11566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:47.042675 11566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20efe0fd88104e47903095aed754ebc4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43439 } }
I20260619 16:46:47.043104 11566 leader_election.cc:302] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20efe0fd88104e47903095aed754ebc4; no voters:
I20260619 16:46:47.043416 11566 leader_election.cc:288] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:47.043532 11571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:47.043740 11571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [term 1 LEADER]: Becoming Leader. State: Replica: 20efe0fd88104e47903095aed754ebc4, State: Running, Role: LEADER
I20260619 16:46:47.043850 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.030727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.3.233.190:43439)
0619 16:46:47.031213 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.031228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.031263 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:46:47.031877 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.031894 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.032281 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.032627 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.033413 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.034438 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.035437 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.035914 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.036089 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.038680 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.038686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.038707 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.038732 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.040508 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.040982 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.040988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.041002 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.041144 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.041425 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.041430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.041431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.041967 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.042191 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.042468 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.042772 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":161,"threads_started":1}
I20260619 16:46:47.044049 11571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20efe0fd88104e47903095aed754ebc4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43439 } }
I20260619 16:46:47.044690 11566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:47.045207 11573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20efe0fd88104e47903095aed754ebc4. Latest consensus state: current_term: 1 leader_uuid: "20efe0fd88104e47903095aed754ebc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20efe0fd88104e47903095aed754ebc4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43439 } } }
I20260619 16:46:47.045343 11573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:47.046043 11576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:47.046941 11572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20efe0fd88104e47903095aed754ebc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20efe0fd88104e47903095aed754ebc4" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43439 } } }
I20260619 16:46:47.047034 11572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:47.049239 11576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:47.054418 11576 catalog_manager.cc:1347] Generated new cluster ID: 289b68e5fdfd45d0a8bcc39b000fc18a
I20260619 16:46:47.054487 11576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:47.064584 11576 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:47.065896 11576 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:47.070585 11576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Generated new TSK 0
I20260619 16:46:47.071444 11576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab.
Entry for principal oryx/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab.
I20260619 16:46:47.101969 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:43439
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab
--principal=oryx/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:47.216835 11592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:47.217120 11592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:47.217166 11592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:47.221284 11592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:47.221352 11592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:47.221392 11592 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:47.221427 11592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:47.227216 11592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab
--principal=oryx/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:47.228664 11592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:47.229700 11592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:47.239426 11598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:47.239697 11592 server_base.cc:1034] running on GCE node
W20260619 16:46:47.239426 11597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:47.239492 11600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:47.240195 11592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:47.240871 11592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:47.242085 11592 hybrid_clock.cc:648] HybridClock initialized: now 1781887607242062 us; error 37 us; skew 500 ppm
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:47.245218 11592 init.cc:377] Logged in from keytab as oryx/127.3.233.129@KRBTEST.COM (short username oryx)
I20260619 16:46:47.246227 11592 webserver.cc:460] Webserver started at http://127.3.233.129:46343/ using document root <none> and password file <none>
I20260619 16:46:47.246870 11592 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:47.246990 11592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:47.247236 11592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:47.249362 11592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "00e37db1a6344da68217d7d7374c8503"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "b741a5bcab0e85f4859162fefcb809cb"
server_key_iv: "01d7f714a4029a887f1672923ca67610"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.249996 11592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "00e37db1a6344da68217d7d7374c8503"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "b741a5bcab0e85f4859162fefcb809cb"
server_key_iv: "01d7f714a4029a887f1672923ca67610"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.253858 11592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 16:46:47.256498 11607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:47.257740 11592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:47.257880 11592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "00e37db1a6344da68217d7d7374c8503"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "b741a5bcab0e85f4859162fefcb809cb"
server_key_iv: "01d7f714a4029a887f1672923ca67610"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.257998 11592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:47.272156 11592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:47.275648 11592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:47.275867 11592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:47.277393 11592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:47.277474 11592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:47.277541 11592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:47.277593 11592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:47.288897 11592 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:34557
I20260619 16:46:47.288966 11722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:34557 every 8 connection(s)
I20260619 16:46:47.290050 11592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:47.292727 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.278472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:43119 (local address 127.3.233.190:43439)
0619 16:46:47.278626 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.278630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.279744 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 16:46:47.280877 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.280883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.280947 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.281006 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.282902 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.283856 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.285715 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.285993 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.286120 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.289008 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.289013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.289016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.289045 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.290810 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.291378 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.291381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.291382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.291487 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.291759 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.291760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.291761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.291923 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.292007 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.292502 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.292610 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:46:47.293522 11616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.279065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43439 (local address 127.3.233.129:43119)
0619 16:46:47.279583 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.279628 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.280623 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.281497 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.281508 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.281971 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:47.282709 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.282730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.283998 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.284002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.285556 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.285569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.286160 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.286164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.286352 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.287046 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:47.287068 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:47.288795 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:47.290939 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.290945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.290960 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.291292 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.291568 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.291570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.291571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.291699 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.292051 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:47.292056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:47.292272 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.292416 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.292668 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":142,"threads_started":1}
I20260619 16:46:47.294798 11723 heartbeater.cc:347] Connected to a master server at 127.3.233.190:43439
I20260619 16:46:47.294952 11723 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:47.295511 11723 heartbeater.cc:511] Master 127.3.233.190:43439 requested a full tablet report, sending...
I20260619 16:46:47.297065 11520 ts_manager.cc:194] Registered new tserver with Master: 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557)
I20260619 16:46:47.298193 11520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:43119
I20260619 16:46:47.298866 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 11592
I20260619 16:46:47.299055 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:47.299437 4006 external_mini_cluster.cc:1428] Setting key 9d6b8f968124afdeafbb48d4d69223e1
I20260619 16:46:47.304486 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.297003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43439 (local address 127.3.233.129:47153)
0619 16:46:47.297350 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.297362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.297500 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.297743 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.297746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.297932 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:47.298023 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.298028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.298783 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.298786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.299689 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.299699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.300327 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.300330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.300413 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.300948 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:47.300962 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:47.301292 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:47.303073 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.303076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.303078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.303269 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.303566 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.303568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.303570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.303648 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.304090 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:47.304092 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:47.304159 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.304236 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.304335 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260619 16:46:47.304662 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.297116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:47153 (local address 127.3.233.190:43439)
0619 16:46:47.297230 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.297233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.297434 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:46:47.297599 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.297602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.297648 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.297729 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.298127 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.298665 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.299822 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.300116 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.300186 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.301420 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.301424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.301426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.301449 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.302876 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.303368 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.303375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.303379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.303463 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.303748 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.303754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.303759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.303999 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.304100 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.304315 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.304436 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.3.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab.
Entry for principal oryx/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab.
I20260619 16:46:47.326025 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:43439
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab
--principal=oryx/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:47.436110 11732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:47.436348 11732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:47.436391 11732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:47.440459 11732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:47.440519 11732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:47.440559 11732 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:47.440625 11732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:47.446375 11732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab
--principal=oryx/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:47.447548 11732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:47.448559 11732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:47.458127 11738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:47.458078 11737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:47.458072 11740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:47.458267 11732 server_base.cc:1034] running on GCE node
I20260619 16:46:47.458740 11732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:47.459399 11732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:47.460578 11732 hybrid_clock.cc:648] HybridClock initialized: now 1781887607460547 us; error 44 us; skew 500 ppm
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:47.463547 11732 init.cc:377] Logged in from keytab as oryx/127.3.233.130@KRBTEST.COM (short username oryx)
I20260619 16:46:47.464392 11732 webserver.cc:460] Webserver started at http://127.3.233.130:35207/ using document root <none> and password file <none>
I20260619 16:46:47.464933 11732 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:47.464993 11732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:47.465152 11732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:47.467167 11732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "8bf03660bbd040468d8fba78d431ef44"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "d7c47d3d72b6ec4f30ff6e4e494bd919"
server_key_iv: "a8a7695ef33415db54334e34e3722441"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.467676 11732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "8bf03660bbd040468d8fba78d431ef44"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "d7c47d3d72b6ec4f30ff6e4e494bd919"
server_key_iv: "a8a7695ef33415db54334e34e3722441"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.471341 11732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:46:47.473945 11747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:47.475400 11732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:46:47.475543 11732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "8bf03660bbd040468d8fba78d431ef44"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "d7c47d3d72b6ec4f30ff6e4e494bd919"
server_key_iv: "a8a7695ef33415db54334e34e3722441"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.475662 11732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:47.490176 11732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:47.493541 11732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:47.493765 11732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:47.495374 11732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:47.495460 11732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:47.495530 11732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:47.495561 11732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:47.507194 11732 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:41143
I20260619 16:46:47.507501 11862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:41143 every 8 connection(s)
I20260619 16:46:47.508289 11732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:47.510147 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.496215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:33005 (local address 127.3.233.190:43439)
0619 16:46:47.496378 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.496385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.497221 (+ 836us) server_negotiation.cc:407] Connection header received
0619 16:46:47.498234 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.498241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.498327 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.498398 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.500225 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.501012 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.502497 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.502787 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.502951 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.506244 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.506248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.506249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.506275 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.507934 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.508533 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.508536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.508537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.508611 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.509089 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.509094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.509097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.509303 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.509380 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.509875 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.510028 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:46:47.510851 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.496601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43439 (local address 127.3.233.130:33005)
0619 16:46:47.497042 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.497091 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.497978 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.498884 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.498897 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.499352 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:47.500063 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.500086 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.501154 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.501158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.502379 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.502388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.503015 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.503020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.503219 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.503903 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:47.503919 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:47.506047 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:47.508050 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.508055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.508069 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.508404 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.508744 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.508747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.508749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.508964 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.509406 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:47.509411 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:47.509631 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.509783 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.510037 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":132,"threads_started":1}
I20260619 16:46:47.512145 11863 heartbeater.cc:347] Connected to a master server at 127.3.233.190:43439
I20260619 16:46:47.512320 11863 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:47.512449 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 11732
I20260619 16:46:47.512586 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:47.512957 4006 external_mini_cluster.cc:1428] Setting key fdee5717589cc6651ad544646361f333
I20260619 16:46:47.512931 11863 heartbeater.cc:511] Master 127.3.233.190:43439 requested a full tablet report, sending...
I20260619 16:46:47.513880 11520 ts_manager.cc:194] Registered new tserver with Master: 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143)
I20260619 16:46:47.514614 11520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:33005
I20260619 16:46:47.522827 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.514385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:40521 (local address 127.3.233.190:43439)
0619 16:46:47.514495 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.514516 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.515260 (+ 744us) server_negotiation.cc:407] Connection header received
0619 16:46:47.515425 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.515429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.515482 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.515575 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.516053 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.516598 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.517854 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.518070 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.518155 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.519488 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.519491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.519494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.519554 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.521039 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.521528 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.521531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.521532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.521598 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.521910 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.521913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.521914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.522174 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.522258 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.522582 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.522713 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:46:47.522989 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.514299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43439 (local address 127.3.233.130:40521)
0619 16:46:47.515175 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.515188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.515300 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.515590 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.515593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.515789 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:47.515884 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.515889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.516728 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.516730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.517723 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.517736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.518188 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.518195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.518320 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.518827 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:47.518847 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:47.519354 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:47.521152 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.521155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.521158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.521418 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.521684 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.521690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.521693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.521796 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.522284 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:47.522287 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:47.522395 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.522526 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.522684 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab.
Entry for principal oryx/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab.
I20260619 16:46:47.539986 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:43439
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab
--principal=oryx/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:47.649886 11872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:47.650161 11872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:47.650206 11872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:47.654094 11872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:47.654191 11872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:47.654263 11872 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:46:47.654351 11872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:47.659667 11872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab
--principal=oryx/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.233.190:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:47.660748 11872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:47.661803 11872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:47.671586 11877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:47.671627 11872 server_base.cc:1034] running on GCE node
W20260619 16:46:47.671581 11880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:47.671581 11878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:47.672279 11872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:47.672941 11872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:47.674144 11872 hybrid_clock.cc:648] HybridClock initialized: now 1781887607674108 us; error 48 us; skew 500 ppm
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:47.677229 11872 init.cc:377] Logged in from keytab as oryx/127.3.233.131@KRBTEST.COM (short username oryx)
I20260619 16:46:47.678164 11872 webserver.cc:460] Webserver started at http://127.3.233.131:41655/ using document root <none> and password file <none>
I20260619 16:46:47.678808 11872 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:47.678896 11872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:47.679152 11872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:47.681286 11872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "b1e23afbdf574930ac825d79e899ed18"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "e37088c4a63bb17442b3683f11c2713d"
server_key_iv: "148e0ca92a3b814f2962ea09fba92591"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.681891 11872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "b1e23afbdf574930ac825d79e899ed18"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "e37088c4a63bb17442b3683f11c2713d"
server_key_iv: "148e0ca92a3b814f2962ea09fba92591"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.686672 11872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260619 16:46:47.689088 11887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:47.690299 11872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:47.690423 11872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "b1e23afbdf574930ac825d79e899ed18"
format_stamp: "Formatted at 2026-06-19 16:46:47 on dist-test-slave-cgwx"
server_key: "e37088c4a63bb17442b3683f11c2713d"
server_key_iv: "148e0ca92a3b814f2962ea09fba92591"
server_key_version: "encryptionkey@0"
I20260619 16:46:47.690580 11872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:47.711210 11872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:47.714699 11872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:47.714900 11872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:47.716383 11872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:47.716454 11872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:47.716521 11872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:47.716568 11872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:47.727365 11872 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:38937
I20260619 16:46:47.727393 12002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:38937 every 8 connection(s)
I20260619 16:46:47.728308 11872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:47.731705 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.717152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:32871 (local address 127.3.233.190:43439)
0619 16:46:47.717274 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.717278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.718410 (+ 1132us) server_negotiation.cc:407] Connection header received
0619 16:46:47.719584 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.719588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.719651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.719745 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.721371 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.722297 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.724257 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.724513 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.724599 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.727787 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.727791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.727794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.727829 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.729531 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.730114 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.730117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.730119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.730205 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.730554 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.730558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.730559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.730749 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.730850 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.731421 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.731542 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:46:47.732482 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.717730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43439 (local address 127.3.233.131:32871)
0619 16:46:47.718248 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.718290 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.719329 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.719964 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.719976 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.720485 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:47.721192 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.721214 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.722447 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.722453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.724107 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.724123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.724693 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.724698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.724908 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.725676 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:47.725704 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:47.727605 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:47.729648 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.729654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.729669 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.730001 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.730299 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.730302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.730304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.730447 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.730861 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:47.730869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:47.731182 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.731364 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.731688 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":136,"threads_started":1}
I20260619 16:46:47.733747 12003 heartbeater.cc:347] Connected to a master server at 127.3.233.190:43439
I20260619 16:46:47.733886 12003 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:47.734460 12003 heartbeater.cc:511] Master 127.3.233.190:43439 requested a full tablet report, sending...
I20260619 16:46:47.735571 11515 ts_manager.cc:194] Registered new tserver with Master: b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937)
I20260619 16:46:47.736227 11515 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:32871
I20260619 16:46:47.736598 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 11872
I20260619 16:46:47.736742 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:47.737079 4006 external_mini_cluster.cc:1428] Setting key c95aa2ee8c119b5e689942153be85b17
I20260619 16:46:47.739099 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:46:47.743901 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.736241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:48663 (local address 127.3.233.190:43439)
0619 16:46:47.736366 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.736371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.736549 (+ 178us) server_negotiation.cc:407] Connection header received
0619 16:46:47.736688 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.736692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.736741 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.736833 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.737436 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.738186 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.739239 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.739439 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.739554 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.740670 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.740674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.740678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.740704 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.742186 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.742656 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.742659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.742660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.742729 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.743094 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.743097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.743099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.743269 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.743375 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.743639 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.743748 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:46:47.743911 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.736108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43439 (local address 127.3.233.131:48663)
0619 16:46:47.736452 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.736466 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.736588 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.736964 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.736968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.737179 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:47.737309 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.737317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.738371 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.738374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.739092 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.739102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.739562 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.739565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.739653 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.740202 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:47.740213 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:47.740525 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:47.742316 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.742321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.742323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.742524 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.742843 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:47.742847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:47.742849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:47.742992 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:47.743365 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:47.743368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:47.743467 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.743569 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.743711 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:46:47.764770 11569 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:47.757658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.3.233.190:43439)
0619 16:46:47.757781 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.757785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.757797 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:46:47.757836 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.757838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.757875 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.757951 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.758714 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.759523 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.760359 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.760575 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.760659 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.764603 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54948: BlockingRecv error: recv got EOF from 127.0.0.1:54948 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:46:47 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:47.774717 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.766185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.3.233.190:43439)
0619 16:46:47.766315 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.766319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.766527 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:46:47.766639 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.766642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.766684 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.766759 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:47.767391 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.767900 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.768673 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.768870 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.768944 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.771478 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:47.771481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:47.771483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:47.771506 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:47.773125 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.773557 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.773560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.773561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.773638 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:47.773906 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:47.773908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:47.773909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:47.774138 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:47.774244 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.774497 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.774597 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:46:47.777828 11515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:47.781106 11515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:47.796734 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.790225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:41143 (local address 127.0.0.1:46602)
0619 16:46:47.790661 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.790710 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.790848 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.792104 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.792110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.792132 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:47.792288 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.792296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.793570 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.793574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.795066 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.795076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.796171 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.796177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.796290 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.796332 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.796446 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.796583 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":77,"threads_started":1}
I20260619 16:46:47.796752 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.791171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.3.233.130:41143)
0619 16:46:47.791560 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.791575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.791602 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:46:47.791683 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.791688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.791911 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.792445 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:47.792588 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.793457 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.795237 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.796046 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.796153 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.796291 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.796510 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.796607 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":96,"threads_started":1}
I20260619 16:46:47.797159 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.790402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.3.233.131:38937)
0619 16:46:47.790863 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.790879 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.791845 (+ 966us) server_negotiation.cc:407] Connection header received
0619 16:46:47.792081 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.792089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.792283 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.792411 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:47.792925 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.793702 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.795289 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.795946 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.796032 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.796189 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.796573 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.796665 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":100,"threads_started":1}
I20260619 16:46:47.797552 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.790233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38937 (local address 127.0.0.1:56150)
0619 16:46:47.791742 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.791758 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.791920 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.792597 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.792600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.792625 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:47.792793 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.792798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.793817 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.793821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.795149 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.795160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.796298 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.796300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.796426 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.796452 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.797368 (+ 916us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.797445 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"mutex_wait_us":50,"spinlock_wait_cycles":1408,"thread_start_us":138,"threads_started":1}
I20260619 16:46:47.797978 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.791167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34557 (local address 127.0.0.1:48596)
0619 16:46:47.792039 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.792051 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.792126 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.792592 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.792594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.792623 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:47.792794 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.792800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.795251 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.795254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.796369 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.796377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.797593 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.797599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.797698 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.797721 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.797789 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.797849 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":61,"spinlock_wait_cycles":2304,"thread_start_us":96,"threads_started":1}
I20260619 16:46:47.799250 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.791473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.3.233.129:34557)
0619 16:46:47.791887 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.791902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.792207 (+ 305us) server_negotiation.cc:407] Connection header received
0619 16:46:47.792277 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.792282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.792457 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.792597 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:47.793843 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.795083 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.796537 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.797322 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.797416 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.797532 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.798217 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.798308 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":103,"threads_started":1}
I20260619 16:46:47.800011 11797 tablet_service.cc:1467] Processing CreateTablet for tablet 461b17be4e834a9a9283592601428e03 (DEFAULT_TABLE table=test-table [id=d94a148417364c5fbf0517a7f6378be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:47.800100 11937 tablet_service.cc:1467] Processing CreateTablet for tablet 461b17be4e834a9a9283592601428e03 (DEFAULT_TABLE table=test-table [id=d94a148417364c5fbf0517a7f6378be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:47.801149 11937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461b17be4e834a9a9283592601428e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:47.801147 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461b17be4e834a9a9283592601428e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:47.801712 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 461b17be4e834a9a9283592601428e03 (DEFAULT_TABLE table=test-table [id=d94a148417364c5fbf0517a7f6378be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:47.802676 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461b17be4e834a9a9283592601428e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:47.808791 12031 tablet_bootstrap.cc:492] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Bootstrap starting.
I20260619 16:46:47.809721 12033 tablet_bootstrap.cc:492] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Bootstrap starting.
I20260619 16:46:47.810098 12032 tablet_bootstrap.cc:492] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Bootstrap starting.
I20260619 16:46:47.811160 12031 tablet_bootstrap.cc:654] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:47.811882 12033 tablet_bootstrap.cc:654] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:47.811959 12032 tablet_bootstrap.cc:654] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:47.812000 12031 log.cc:824] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:47.812690 12032 log.cc:824] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:47.812690 12033 log.cc:824] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:47.814440 12031 tablet_bootstrap.cc:492] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: No bootstrap required, opened a new log
I20260619 16:46:47.814594 12032 tablet_bootstrap.cc:492] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: No bootstrap required, opened a new log
I20260619 16:46:47.814613 12033 tablet_bootstrap.cc:492] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: No bootstrap required, opened a new log
I20260619 16:46:47.814666 12031 ts_tablet_manager.cc:1397] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:46:47.814776 12033 ts_tablet_manager.cc:1397] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:46:47.814776 12032 ts_tablet_manager.cc:1397] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 16:46:47.817749 12031 raft_consensus.cc:348] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.817996 12031 raft_consensus.cc:374] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:47.818050 12031 raft_consensus.cc:729] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e37db1a6344da68217d7d7374c8503, State: Initialized, Role: FOLLOWER
I20260619 16:46:47.818500 12031 consensus_queue.cc:260] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.818732 12032 raft_consensus.cc:348] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.818732 12033 raft_consensus.cc:348] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.819007 12032 raft_consensus.cc:374] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:47.819072 12032 raft_consensus.cc:729] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e23afbdf574930ac825d79e899ed18, State: Initialized, Role: FOLLOWER
I20260619 16:46:47.819207 11723 heartbeater.cc:503] Master 127.3.233.190:43439 was elected leader, sending a full tablet report...
I20260619 16:46:47.819523 12031 ts_tablet_manager.cc:1428] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:46:47.819460 12032 consensus_queue.cc:260] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.819022 12033 raft_consensus.cc:374] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:47.819654 12033 raft_consensus.cc:729] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf03660bbd040468d8fba78d431ef44, State: Initialized, Role: FOLLOWER
I20260619 16:46:47.820072 12033 consensus_queue.cc:260] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.820458 12032 ts_tablet_manager.cc:1428] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:46:47.820770 12003 heartbeater.cc:503] Master 127.3.233.190:43439 was elected leader, sending a full tablet report...
I20260619 16:46:47.821697 12033 ts_tablet_manager.cc:1428] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260619 16:46:47.821789 11863 heartbeater.cc:503] Master 127.3.233.190:43439 was elected leader, sending a full tablet report...
I20260619 16:46:47.864427 12039 raft_consensus.cc:482] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:47.864687 12039 raft_consensus.cc:504] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.865967 12039 leader_election.cc:288] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937), 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557)
I20260619 16:46:47.870191 12027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.866296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:56801 (local address 127.3.233.131:38937)
0619 16:46:47.866420 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.866425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.866443 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:47.866607 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.866610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.866648 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.866727 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:47.867099 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.867693 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.869024 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.869632 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.869750 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.869806 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.870018 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.870094 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:46:47.870208 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.866196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38937 (local address 127.3.233.130:56801)
0619 16:46:47.866360 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.866380 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.866477 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.866749 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.866752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.866797 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:47.866911 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.866916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.867813 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.867817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.868894 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.868904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.869747 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.869750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.869892 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.869917 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.869997 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.870069 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":16,"spinlock_wait_cycles":7808}
I20260619 16:46:47.871022 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.866390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:44713 (local address 127.3.233.129:34557)
0619 16:46:47.866522 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.866526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.866738 (+ 212us) server_negotiation.cc:407] Connection header received
0619 16:46:47.866952 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.866959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.867014 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.867143 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:47.867409 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.868010 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.869899 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.870475 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.870577 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.870611 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.870822 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.870894 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:46:47.871042 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.866272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34557 (local address 127.3.233.130:44713)
0619 16:46:47.866639 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:47.866650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:47.866815 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:47.867160 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:47.867163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:47.867179 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:47.867307 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.867311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.868158 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.868161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.869776 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:47.869785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.870595 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.870599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:47.870695 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.870713 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:46:47.870804 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:46:47.870866 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":1,"spinlock_wait_cycles":48896,"thread_start_us":124,"threads_started":1}
I20260619 16:46:47.871227 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461b17be4e834a9a9283592601428e03" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18" is_pre_election: true
I20260619 16:46:47.871549 11957 raft_consensus.cc:2454] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 0.
I20260619 16:46:47.872188 11749 leader_election.cc:302] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf03660bbd040468d8fba78d431ef44, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:47.872273 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461b17be4e834a9a9283592601428e03" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e37db1a6344da68217d7d7374c8503" is_pre_election: true
I20260619 16:46:47.872489 12039 raft_consensus.cc:2781] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:47.872560 11677 raft_consensus.cc:2454] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 0.
I20260619 16:46:47.872604 12039 raft_consensus.cc:482] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:47.872668 12039 raft_consensus.cc:3037] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:47.873837 12039 raft_consensus.cc:504] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.874193 12039 leader_election.cc:288] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 election: Requested vote from peers b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937), 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557)
I20260619 16:46:47.874603 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461b17be4e834a9a9283592601428e03" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18"
I20260619 16:46:47.874675 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "461b17be4e834a9a9283592601428e03" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e37db1a6344da68217d7d7374c8503"
I20260619 16:46:47.874738 11957 raft_consensus.cc:3037] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:47.874781 11677 raft_consensus.cc:3037] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:47.876099 11677 raft_consensus.cc:2454] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 1.
I20260619 16:46:47.876099 11957 raft_consensus.cc:2454] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 1.
I20260619 16:46:47.876446 11749 leader_election.cc:302] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf03660bbd040468d8fba78d431ef44, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:47.876703 12039 raft_consensus.cc:2781] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:47.876902 12039 raft_consensus.cc:686] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 1 LEADER]: Becoming Leader. State: Replica: 8bf03660bbd040468d8fba78d431ef44, State: Running, Role: LEADER
I20260619 16:46:47.877338 12039 consensus_queue.cc:237] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } }
I20260619 16:46:47.880740 11515 catalog_manager.cc:5495] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130). New cstate: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:47.896956 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:47.892798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.3.233.130:41143)
0619 16:46:47.892962 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:47.892996 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:47.893013 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:47.893332 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:47.893334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:47.893414 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:47.893571 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:47.893811 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.894335 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.895390 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:47.895690 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:47.895778 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:47.896073 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:47.896494 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:47.896609 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:47.896699 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 16:46:47.896752 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:46:47.903517 11677 raft_consensus.cc:1261] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Refusing update from remote peer 8bf03660bbd040468d8fba78d431ef44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:47.903831 11957 raft_consensus.cc:1261] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Refusing update from remote peer 8bf03660bbd040468d8fba78d431ef44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:47.904349 12041 consensus_queue.cc:1035] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:47.904636 12039 consensus_queue.cc:1035] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:47.919689 11515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:47.919872 11515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:47.922180 11515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Sending DeleteTablet for 3 replicas of tablet 461b17be4e834a9a9283592601428e03
I20260619 16:46:47.922884 11937 tablet_service.cc:1514] Processing DeleteTablet for tablet 461b17be4e834a9a9283592601428e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:47 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260619 16:46:47.923050 11797 tablet_service.cc:1514] Processing DeleteTablet for tablet 461b17be4e834a9a9283592601428e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:47 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260619 16:46:47.923125 11657 tablet_service.cc:1514] Processing DeleteTablet for tablet 461b17be4e834a9a9283592601428e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:47 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:48596
I20260619 16:46:47.923561 12055 tablet_replica.cc:330] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: stopping tablet replica
I20260619 16:46:47.923679 12056 tablet_replica.cc:330] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: stopping tablet replica
I20260619 16:46:47.923873 12055 raft_consensus.cc:2229] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:47.923892 12056 raft_consensus.cc:2229] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:47.924190 12055 raft_consensus.cc:2258] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:47.924268 11515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:47.924659 11515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:47.925405 12056 raft_consensus.cc:2258] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:47.925743 12055 ts_tablet_manager.cc:1905] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:47.928508 12056 ts_tablet_manager.cc:1905] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:47.928541 12055 ts_tablet_manager.cc:1918] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:47.928622 12055 log.cc:1196] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/461b17be4e834a9a9283592601428e03
I20260619 16:46:47.928936 12055 ts_tablet_manager.cc:1939] T 461b17be4e834a9a9283592601428e03 P 8bf03660bbd040468d8fba78d431ef44: Deleting consensus metadata
I20260619 16:46:47.929661 11937 tablet_service.cc:1467] Processing CreateTablet for tablet f8ceca99a28d4e1e84d0f26f4cd8af77 (DEFAULT_TABLE table=test-table [id=fd014eb00577496085eab18272e251e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:47.929750 11797 tablet_service.cc:1467] Processing CreateTablet for tablet f8ceca99a28d4e1e84d0f26f4cd8af77 (DEFAULT_TABLE table=test-table [id=fd014eb00577496085eab18272e251e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:47.929857 11657 tablet_service.cc:1467] Processing CreateTablet for tablet f8ceca99a28d4e1e84d0f26f4cd8af77 (DEFAULT_TABLE table=test-table [id=fd014eb00577496085eab18272e251e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:47.929924 11937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ceca99a28d4e1e84d0f26f4cd8af77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:47.929978 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ceca99a28d4e1e84d0f26f4cd8af77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:47.930090 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8ceca99a28d4e1e84d0f26f4cd8af77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:47.932256 12032 tablet_bootstrap.cc:492] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Bootstrap starting.
I20260619 16:46:47.932683 12031 tablet_bootstrap.cc:492] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Bootstrap starting.
I20260619 16:46:47.933307 12032 tablet_bootstrap.cc:654] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:47.933477 11504 catalog_manager.cc:4841] TS 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143): tablet 461b17be4e834a9a9283592601428e03 (table test-table [id=d94a148417364c5fbf0517a7f6378be7]) successfully deleted
I20260619 16:46:47.933707 12031 tablet_bootstrap.cc:654] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:47.933991 12033 tablet_bootstrap.cc:492] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Bootstrap starting.
I20260619 16:46:47.934628 12032 tablet_bootstrap.cc:492] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: No bootstrap required, opened a new log
I20260619 16:46:47.934712 12032 ts_tablet_manager.cc:1397] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 16:46:47.934989 12033 tablet_bootstrap.cc:654] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:47.935218 12032 raft_consensus.cc:348] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:47.935343 12032 raft_consensus.cc:374] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:47.935374 12032 raft_consensus.cc:729] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e23afbdf574930ac825d79e899ed18, State: Initialized, Role: FOLLOWER
I20260619 16:46:47.935712 12032 consensus_queue.cc:260] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:47.935770 12056 ts_tablet_manager.cc:1918] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:47.935850 12056 log.cc:1196] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/461b17be4e834a9a9283592601428e03
I20260619 16:46:47.936184 12056 ts_tablet_manager.cc:1939] T 461b17be4e834a9a9283592601428e03 P 00e37db1a6344da68217d7d7374c8503: Deleting consensus metadata
I20260619 16:46:47.936646 12032 ts_tablet_manager.cc:1428] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:47.937364 11504 catalog_manager.cc:4841] TS 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557): tablet 461b17be4e834a9a9283592601428e03 (table test-table [id=d94a148417364c5fbf0517a7f6378be7]) successfully deleted
I20260619 16:46:47.937412 12054 tablet_replica.cc:330] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: stopping tablet replica
I20260619 16:46:47.937728 12054 raft_consensus.cc:2229] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:47.938078 12054 raft_consensus.cc:2258] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:47.939055 12031 tablet_bootstrap.cc:492] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: No bootstrap required, opened a new log
I20260619 16:46:47.939152 12031 ts_tablet_manager.cc:1397] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 16:46:47.939738 12031 raft_consensus.cc:348] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:47.939838 12031 raft_consensus.cc:374] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:47.939868 12031 raft_consensus.cc:729] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e37db1a6344da68217d7d7374c8503, State: Initialized, Role: FOLLOWER
I20260619 16:46:47.939985 12031 consensus_queue.cc:260] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:47.940075 12054 ts_tablet_manager.cc:1905] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:47.940222 12031 ts_tablet_manager.cc:1428] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:47.941291 12033 tablet_bootstrap.cc:492] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: No bootstrap required, opened a new log
I20260619 16:46:47.941403 12033 ts_tablet_manager.cc:1397] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260619 16:46:47.941813 12033 raft_consensus.cc:348] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:47.941929 12033 raft_consensus.cc:374] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:47.941968 12033 raft_consensus.cc:729] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf03660bbd040468d8fba78d431ef44, State: Initialized, Role: FOLLOWER
I20260619 16:46:47.942111 12033 consensus_queue.cc:260] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:47.942363 12033 ts_tablet_manager.cc:1428] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:47.943076 12054 ts_tablet_manager.cc:1918] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:47.943177 12054 log.cc:1196] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/461b17be4e834a9a9283592601428e03
I20260619 16:46:47.943507 12054 ts_tablet_manager.cc:1939] T 461b17be4e834a9a9283592601428e03 P b1e23afbdf574930ac825d79e899ed18: Deleting consensus metadata
I20260619 16:46:47.944728 11505 catalog_manager.cc:4841] TS b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937): tablet 461b17be4e834a9a9283592601428e03 (table test-table [id=d94a148417364c5fbf0517a7f6378be7]) successfully deleted
W20260619 16:46:47.980223 12004 tablet.cc:2369] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:48.009567 11864 tablet.cc:2369] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:46:48.041610 11724 tablet.cc:2369] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:46:48.148598 12038 raft_consensus.cc:482] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:48.148757 12038 raft_consensus.cc:504] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.150043 12038 leader_election.cc:288] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557), 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143)
I20260619 16:46:48.154392 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.150347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34557 (local address 127.3.233.131:38243)
0619 16:46:48.150490 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.150506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.150613 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.150957 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.150960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.151052 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:48.151203 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.151210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.152189 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.152192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.153079 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.153088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.153908 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.153912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.154072 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.154099 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.154178 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.154250 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:46:48.154484 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.150506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:38243 (local address 127.3.233.129:34557)
0619 16:46:48.150657 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.150662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.150684 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:46:48.150749 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.150754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.150828 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.150928 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:48.151347 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.152072 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.153216 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.153772 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.153872 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.153913 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.154267 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.154347 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:46:48.154762 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.150809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:58817 (local address 127.3.233.130:41143)
0619 16:46:48.150967 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.150971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.151134 (+ 163us) server_negotiation.cc:407] Connection header received
0619 16:46:48.151244 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.151249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.151309 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.151381 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:48.151729 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.152292 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.153575 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.154090 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.154213 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.154257 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.154567 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.154657 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:46:48.154994 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ceca99a28d4e1e84d0f26f4cd8af77" candidate_uuid: "b1e23afbdf574930ac825d79e899ed18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e37db1a6344da68217d7d7374c8503" is_pre_election: true
I20260619 16:46:48.155103 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.150673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:41143 (local address 127.3.233.131:58817)
0619 16:46:48.151029 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.151045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.151151 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.151422 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.151426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.151445 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:48.151607 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.151614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.152407 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.152410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.153433 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.153444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.154275 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.154278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.154382 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.154404 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.154900 (+ 496us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.154998 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":98,"threads_started":1}
I20260619 16:46:48.155176 11677 raft_consensus.cc:2454] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e23afbdf574930ac825d79e899ed18 in term 0.
I20260619 16:46:48.155503 11888 leader_election.cc:302] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e37db1a6344da68217d7d7374c8503, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:48.155767 12038 raft_consensus.cc:2781] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:48.155942 12038 raft_consensus.cc:482] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:48.155988 12038 raft_consensus.cc:3037] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.156162 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ceca99a28d4e1e84d0f26f4cd8af77" candidate_uuid: "b1e23afbdf574930ac825d79e899ed18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf03660bbd040468d8fba78d431ef44" is_pre_election: true
I20260619 16:46:48.156394 11817 raft_consensus.cc:2454] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1e23afbdf574930ac825d79e899ed18 in term 0.
I20260619 16:46:48.157138 12038 raft_consensus.cc:504] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.157491 12038 leader_election.cc:288] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [CANDIDATE]: Term 1 election: Requested vote from peers 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557), 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143)
I20260619 16:46:48.157840 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ceca99a28d4e1e84d0f26f4cd8af77" candidate_uuid: "b1e23afbdf574930ac825d79e899ed18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e37db1a6344da68217d7d7374c8503"
I20260619 16:46:48.157898 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8ceca99a28d4e1e84d0f26f4cd8af77" candidate_uuid: "b1e23afbdf574930ac825d79e899ed18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf03660bbd040468d8fba78d431ef44"
I20260619 16:46:48.157958 11677 raft_consensus.cc:3037] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.157999 11817 raft_consensus.cc:3037] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.159152 11677 raft_consensus.cc:2454] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e23afbdf574930ac825d79e899ed18 in term 1.
I20260619 16:46:48.159158 11817 raft_consensus.cc:2454] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1e23afbdf574930ac825d79e899ed18 in term 1.
I20260619 16:46:48.159415 11888 leader_election.cc:302] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e37db1a6344da68217d7d7374c8503, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:48.159601 12038 raft_consensus.cc:2781] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:48.159792 12038 raft_consensus.cc:686] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 1 LEADER]: Becoming Leader. State: Replica: b1e23afbdf574930ac825d79e899ed18, State: Running, Role: LEADER
I20260619 16:46:48.160051 12038 consensus_queue.cc:237] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.162438 11515 catalog_manager.cc:5495] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1e23afbdf574930ac825d79e899ed18 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "b1e23afbdf574930ac825d79e899ed18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } health_report { overall_health: HEALTHY } } }
Jun 19 16:46:48 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887606, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.190@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:48.203220 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.188796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.3.233.190:43439)
0619 16:46:48.188993 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.188999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.189263 (+ 264us) server_negotiation.cc:407] Connection header received
0619 16:46:48.189432 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.189437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.189487 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.189602 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:48.190291 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.190860 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.191751 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.191985 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.192099 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.199633 (+ 7534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:48.199640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:48.199645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:48.199680 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:48.201331 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:48.201886 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:48.201889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:48.201890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:48.201977 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:48.202253 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:48.202255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:48.202256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:48.202444 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:48.202519 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.202882 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.203082 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:46:48.203231 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.188780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43439 (local address 127.0.0.1:54970)
0619 16:46:48.189148 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.189162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.189325 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.189621 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.189624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.190062 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:48.190156 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.190163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.191028 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.191032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.191630 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.191637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.192100 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.192104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.192189 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.197710 (+ 5521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:48.197725 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:48.199424 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:48.201453 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:48.201461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:48.201467 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:48.201764 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:48.202080 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:48.202083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:48.202084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:48.202168 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:48.202561 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:48.202567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:48.202681 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.202781 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.202999 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":102,"threads_started":1}
I20260619 16:46:48.207849 11515 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:54970:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:46:48.208676 11515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:48.213930 11797 tablet_service.cc:1467] Processing CreateTablet for tablet 64e1d4c8f9314cc0a878bd112090a2f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9458c435e18c4e92b6496ebad857cdd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:48.213935 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 64e1d4c8f9314cc0a878bd112090a2f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9458c435e18c4e92b6496ebad857cdd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:48.214056 11937 tablet_service.cc:1467] Processing CreateTablet for tablet 64e1d4c8f9314cc0a878bd112090a2f8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9458c435e18c4e92b6496ebad857cdd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:46:48.214357 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e1d4c8f9314cc0a878bd112090a2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.214357 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e1d4c8f9314cc0a878bd112090a2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.214488 11937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e1d4c8f9314cc0a878bd112090a2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.217126 12033 tablet_bootstrap.cc:492] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: Bootstrap starting.
I20260619 16:46:48.217128 12032 tablet_bootstrap.cc:492] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: Bootstrap starting.
I20260619 16:46:48.218102 12033 tablet_bootstrap.cc:654] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.218142 12032 tablet_bootstrap.cc:654] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.218693 12031 tablet_bootstrap.cc:492] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: Bootstrap starting.
I20260619 16:46:48.219481 12033 tablet_bootstrap.cc:492] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: No bootstrap required, opened a new log
I20260619 16:46:48.219594 12033 ts_tablet_manager.cc:1397] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 16:46:48.219805 12031 tablet_bootstrap.cc:654] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.219877 12032 tablet_bootstrap.cc:492] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: No bootstrap required, opened a new log
I20260619 16:46:48.219969 12032 ts_tablet_manager.cc:1397] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:48.220093 12033 raft_consensus.cc:348] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.220227 12033 raft_consensus.cc:374] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.220270 12033 raft_consensus.cc:729] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf03660bbd040468d8fba78d431ef44, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.220403 12033 consensus_queue.cc:260] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.220758 12033 ts_tablet_manager.cc:1428] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:48.221072 12031 tablet_bootstrap.cc:492] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: No bootstrap required, opened a new log
I20260619 16:46:48.221168 12031 ts_tablet_manager.cc:1397] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:48.221406 12051 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.221583 12051 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:48.221632 12031 raft_consensus.cc:348] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.221751 12031 raft_consensus.cc:374] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.221776 12031 raft_consensus.cc:729] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e37db1a6344da68217d7d7374c8503, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.221921 12031 consensus_queue.cc:260] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.222205 12031 ts_tablet_manager.cc:1428] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:48.222165 12037 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.222285 12037 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:48.220446 12032 raft_consensus.cc:348] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.223112 12032 raft_consensus.cc:374] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.223166 12032 raft_consensus.cc:729] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e23afbdf574930ac825d79e899ed18, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.223337 12032 consensus_queue.cc:260] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.223598 12038 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.223753 12038 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:48.224056 12032 ts_tablet_manager.cc:1428] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:46:48.445441 12051 raft_consensus.cc:482] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:48.445648 12051 raft_consensus.cc:504] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.446089 12051 leader_election.cc:288] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557), b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937)
I20260619 16:46:48.446580 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1d4c8f9314cc0a878bd112090a2f8" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18" is_pre_election: true
I20260619 16:46:48.446619 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1d4c8f9314cc0a878bd112090a2f8" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e37db1a6344da68217d7d7374c8503" is_pre_election: true
I20260619 16:46:48.446751 11957 raft_consensus.cc:2454] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 0.
I20260619 16:46:48.446801 11677 raft_consensus.cc:2454] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 0.
I20260619 16:46:48.447057 11749 leader_election.cc:302] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf03660bbd040468d8fba78d431ef44, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:48.447252 12051 raft_consensus.cc:2781] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:48.447314 12051 raft_consensus.cc:482] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:48.447352 12051 raft_consensus.cc:3037] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.448448 12051 raft_consensus.cc:504] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.448752 12051 leader_election.cc:288] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 election: Requested vote from peers 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557), b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937)
I20260619 16:46:48.449080 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1d4c8f9314cc0a878bd112090a2f8" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18"
I20260619 16:46:48.449074 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e1d4c8f9314cc0a878bd112090a2f8" candidate_uuid: "8bf03660bbd040468d8fba78d431ef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e37db1a6344da68217d7d7374c8503"
I20260619 16:46:48.449198 11677 raft_consensus.cc:3037] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.449198 11957 raft_consensus.cc:3037] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.450260 11957 raft_consensus.cc:2454] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 1.
I20260619 16:46:48.450260 11677 raft_consensus.cc:2454] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf03660bbd040468d8fba78d431ef44 in term 1.
I20260619 16:46:48.450526 11749 leader_election.cc:302] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf03660bbd040468d8fba78d431ef44, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:48.450722 12051 raft_consensus.cc:2781] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:48.450882 12051 raft_consensus.cc:686] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [term 1 LEADER]: Becoming Leader. State: Replica: 8bf03660bbd040468d8fba78d431ef44, State: Running, Role: LEADER
I20260619 16:46:48.451057 12051 consensus_queue.cc:237] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.451706 12072 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: TxnStatusTablet state changed. Reason: New leader 8bf03660bbd040468d8fba78d431ef44. Latest consensus state: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.451817 12072 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:48.452109 12074 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:46:48.452267 12074 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:46:48.452663 11520 catalog_manager.cc:5495] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130). New cstate: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:48.519887 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.515429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:41143 (local address 127.0.0.1:46624)
0619 16:46:48.515646 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.515661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.515747 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.516033 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.516036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.516044 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:48.516164 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.516170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.517454 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.517458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.518425 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.518435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.519081 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.519085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.519192 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.519246 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.519572 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.519585 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.519663 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.519750 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260619 16:46:48.519886 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.515526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.3.233.130:41143)
0619 16:46:48.515669 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.515674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.515712 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:46:48.515867 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.515870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.515922 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.515999 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:48.516272 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.517154 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.518557 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.518964 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.519073 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.519385 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.519477 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.519546 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.519685 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.519743 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:46:48.522650 11957 raft_consensus.cc:1261] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Refusing update from remote peer 8bf03660bbd040468d8fba78d431ef44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:48.522773 11677 raft_consensus.cc:1261] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Refusing update from remote peer 8bf03660bbd040468d8fba78d431ef44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:48.523118 12072 consensus_queue.cc:1035] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:48.523285 12051 consensus_queue.cc:1035] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:48.524680 12037 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: TxnStatusTablet state changed. Reason: New leader 8bf03660bbd040468d8fba78d431ef44. Latest consensus state: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.524787 12037 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:48.524989 12038 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: TxnStatusTablet state changed. Reason: New leader 8bf03660bbd040468d8fba78d431ef44. Latest consensus state: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.525082 12038 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:48.532317 12051 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.532451 12051 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:48.532606 12072 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.532697 12072 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P 8bf03660bbd040468d8fba78d431ef44: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:46:48.533027 12038 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.533149 12038 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P b1e23afbdf574930ac825d79e899ed18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:48.533128 12037 tablet_replica.cc:439] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bf03660bbd040468d8fba78d431ef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } } }
I20260619 16:46:48.533233 12037 tablet_replica.cc:442] T 64e1d4c8f9314cc0a878bd112090a2f8 P 00e37db1a6344da68217d7d7374c8503: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:46:48.533448 12078 mvcc.cc:205] Tried to move back new op lower bound from 7298611644504678400 to 7298611644216700928. Current Snapshot: MvccSnapshot[applied={T|T < 7298611644504678400}]
I20260619 16:46:48.533829 12079 mvcc.cc:205] Tried to move back new op lower bound from 7298611644504678400 to 7298611644216700928. Current Snapshot: MvccSnapshot[applied={T|T < 7298611644504678400 or (T in {7298611644504678400})}]
I20260619 16:46:48.542703 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.538559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.3.233.131:38937)
0619 16:46:48.538780 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.538784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.538842 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:46:48.539023 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.539026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.539087 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.539220 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:48.539436 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.540211 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.541009 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.541372 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.541495 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.541729 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.542307 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.542429 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.542462 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.542551 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260619 16:46:48.553298 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.550103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:54385 (local address 127.3.233.130:41143)
0619 16:46:48.550309 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.550314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.550365 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:46:48.550530 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.550533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.550590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.550666 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:48.550912 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.551421 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.552262 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.552476 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.552593 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.552867 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.552932 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.553007 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.553115 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.553163 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 16:46:48.553313 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.550012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:41143 (local address 127.3.233.131:54385)
0619 16:46:48.550289 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.550306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.550410 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.550685 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.550688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.550694 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:48.550799 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.550804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.551522 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.551528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.552147 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.552155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.552594 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.552597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.552682 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.552731 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.553021 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.553028 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.553102 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.553178 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
I20260619 16:46:48.559844 11677 raft_consensus.cc:1261] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Refusing update from remote peer b1e23afbdf574930ac825d79e899ed18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:48.559894 11817 raft_consensus.cc:1261] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Refusing update from remote peer b1e23afbdf574930ac825d79e899ed18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:48.560374 12061 consensus_queue.cc:1035] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:48.560554 12038 consensus_queue.cc:1035] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:48.569183 12077 mvcc.cc:205] Tried to move back new op lower bound from 7298611644656427008 to 7298611643025891328. Current Snapshot: MvccSnapshot[applied={T|T < 7298611644656427008}]
I20260619 16:46:48.585343 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.581132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:47463 (local address 127.3.233.131:38937)
0619 16:46:48.581265 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.581270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.581478 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:46:48.581661 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.581664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.581717 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.581791 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:48.582124 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.582871 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.583869 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.584163 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.584275 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.584846 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.584934 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.585008 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.585178 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.585230 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:46:48.585345 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.581085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38937 (local address 127.3.233.130:47463)
0619 16:46:48.581388 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.581402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.581523 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.581816 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.581819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.581832 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:48.582000 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.582005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.583002 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.583005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.583754 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.583764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.584282 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.584286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.584437 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.584683 (+ 246us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.585015 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.585024 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.585103 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.585191 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":103,"threads_started":1}
I20260619 16:46:48.607455 11518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:48.607635 11518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:48.609776 11518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Sending DeleteTablet for 3 replicas of tablet f8ceca99a28d4e1e84d0f26f4cd8af77
I20260619 16:46:48.610420 11657 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ceca99a28d4e1e84d0f26f4cd8af77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:48 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:48596
I20260619 16:46:48.610489 11797 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ceca99a28d4e1e84d0f26f4cd8af77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:48 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260619 16:46:48.610481 11937 tablet_service.cc:1514] Processing DeleteTablet for tablet f8ceca99a28d4e1e84d0f26f4cd8af77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:48 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260619 16:46:48.610911 12101 tablet_replica.cc:330] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: stopping tablet replica
I20260619 16:46:48.611088 12101 raft_consensus.cc:2229] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:48.611223 12101 raft_consensus.cc:2258] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:48.612161 12101 ts_tablet_manager.cc:1905] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:48.611424 12099 tablet_replica.cc:330] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: stopping tablet replica
I20260619 16:46:48.614461 12099 raft_consensus.cc:2229] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:48.614595 12099 raft_consensus.cc:2258] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:48.615411 12099 ts_tablet_manager.cc:1905] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:48.615617 12101 ts_tablet_manager.cc:1918] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:48.615701 12101 log.cc:1196] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/f8ceca99a28d4e1e84d0f26f4cd8af77
I20260619 16:46:48.615986 12101 ts_tablet_manager.cc:1939] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 8bf03660bbd040468d8fba78d431ef44: Deleting consensus metadata
I20260619 16:46:48.616307 12100 tablet_replica.cc:330] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: stopping tablet replica
I20260619 16:46:48.616458 12100 raft_consensus.cc:2229] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:48.616791 12100 raft_consensus.cc:2258] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:48.616829 11504 catalog_manager.cc:4841] TS 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143): tablet f8ceca99a28d4e1e84d0f26f4cd8af77 (table test-table [id=fd014eb00577496085eab18272e251e4]) successfully deleted
I20260619 16:46:48.617512 12100 ts_tablet_manager.cc:1905] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:48.619001 12099 ts_tablet_manager.cc:1918] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:48.619161 12099 log.cc:1196] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/f8ceca99a28d4e1e84d0f26f4cd8af77
I20260619 16:46:48.619515 12099 ts_tablet_manager.cc:1939] T f8ceca99a28d4e1e84d0f26f4cd8af77 P 00e37db1a6344da68217d7d7374c8503: Deleting consensus metadata
I20260619 16:46:48.620352 11504 catalog_manager.cc:4841] TS 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557): tablet f8ceca99a28d4e1e84d0f26f4cd8af77 (table test-table [id=fd014eb00577496085eab18272e251e4]) successfully deleted
I20260619 16:46:48.620883 12100 ts_tablet_manager.cc:1918] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:48.620994 12100 log.cc:1196] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/f8ceca99a28d4e1e84d0f26f4cd8af77
I20260619 16:46:48.621317 12100 ts_tablet_manager.cc:1939] T f8ceca99a28d4e1e84d0f26f4cd8af77 P b1e23afbdf574930ac825d79e899ed18: Deleting consensus metadata
I20260619 16:46:48.622124 11505 catalog_manager.cc:4841] TS b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937): tablet f8ceca99a28d4e1e84d0f26f4cd8af77 (table test-table [id=fd014eb00577496085eab18272e251e4]) successfully deleted
Jun 19 16:46:48 dist-test-slave-cgwx krb5kdc[11475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:46:48 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:48 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:46:48.640695 11569 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:48.633690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.3.233.190:43439)
0619 16:46:48.633838 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.633844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.633998 (+ 154us) server_negotiation.cc:407] Connection header received
0619 16:46:48.634145 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.634148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.634226 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.634327 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:48.635191 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.635715 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.636913 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.637134 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.637215 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.640557 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54972: BlockingRecv error: recv got EOF from 127.0.0.1:54972 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:46:48 dist-test-slave-cgwx krb5kdc[11475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM
I20260619 16:46:48.651110 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.642407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.3.233.190:43439)
0619 16:46:48.642530 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.642534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.642632 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:46:48.642747 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.642749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.642788 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.642852 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:48.643594 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.644091 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.645163 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.645362 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.645437 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.648023 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:48.648027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:48.648030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:48.648059 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:48.649483 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:48.649895 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:48.649899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:48.649901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:48.649981 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:48.650251 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:48.650255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:48.650257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:48.650430 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:48.650562 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.650802 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.650914 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:46:48.653546 11518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:48.653982 11518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:48.658742 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 60501da624ab4e3fa1cafbbdd1675de5 (DEFAULT_TABLE table=test-table [id=98e4f64ce7c441688bd175b9ed566eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:48.658906 11797 tablet_service.cc:1467] Processing CreateTablet for tablet 60501da624ab4e3fa1cafbbdd1675de5 (DEFAULT_TABLE table=test-table [id=98e4f64ce7c441688bd175b9ed566eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:48.658916 11937 tablet_service.cc:1467] Processing CreateTablet for tablet 60501da624ab4e3fa1cafbbdd1675de5 (DEFAULT_TABLE table=test-table [id=98e4f64ce7c441688bd175b9ed566eea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:48.659092 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60501da624ab4e3fa1cafbbdd1675de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.659212 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60501da624ab4e3fa1cafbbdd1675de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.659247 11937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60501da624ab4e3fa1cafbbdd1675de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.661499 12031 tablet_bootstrap.cc:492] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: Bootstrap starting.
I20260619 16:46:48.661505 12033 tablet_bootstrap.cc:492] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: Bootstrap starting.
I20260619 16:46:48.661500 12032 tablet_bootstrap.cc:492] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: Bootstrap starting.
I20260619 16:46:48.662353 12031 tablet_bootstrap.cc:654] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.662353 12033 tablet_bootstrap.cc:654] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.662406 12032 tablet_bootstrap.cc:654] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.663767 12031 tablet_bootstrap.cc:492] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: No bootstrap required, opened a new log
I20260619 16:46:48.663848 12031 ts_tablet_manager.cc:1397] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:48.664340 12031 raft_consensus.cc:348] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.664470 12031 raft_consensus.cc:374] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.664512 12031 raft_consensus.cc:729] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e37db1a6344da68217d7d7374c8503, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.664630 12031 consensus_queue.cc:260] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.664911 12031 ts_tablet_manager.cc:1428] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:46:48.665076 12033 tablet_bootstrap.cc:492] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: No bootstrap required, opened a new log
I20260619 16:46:48.665150 12033 ts_tablet_manager.cc:1397] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 16:46:48.665517 12033 raft_consensus.cc:348] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.665656 12033 raft_consensus.cc:374] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.665690 12033 raft_consensus.cc:729] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf03660bbd040468d8fba78d431ef44, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.665805 12033 consensus_queue.cc:260] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.666046 12033 ts_tablet_manager.cc:1428] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 16:46:48.667426 12032 tablet_bootstrap.cc:492] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: No bootstrap required, opened a new log
I20260619 16:46:48.667549 12032 ts_tablet_manager.cc:1397] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 16:46:48.667935 12032 raft_consensus.cc:348] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.668026 12032 raft_consensus.cc:374] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.668048 12032 raft_consensus.cc:729] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e23afbdf574930ac825d79e899ed18, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.668195 12032 consensus_queue.cc:260] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.668458 12032 ts_tablet_manager.cc:1428] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:48.790685 12037 raft_consensus.cc:482] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:48.790814 12037 raft_consensus.cc:504] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.792294 12037 leader_election.cc:288] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143), b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937)
I20260619 16:46:48.796897 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.792790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:56763 (local address 127.3.233.131:38937)
0619 16:46:48.792951 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.792956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.792971 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:46:48.793169 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.793176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.793214 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.793294 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:48.793570 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.794164 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.795832 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.796343 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.796479 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.796533 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.796706 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.796782 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:46:48.796944 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.792592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38937 (local address 127.3.233.129:56763)
0619 16:46:48.792868 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.792883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.793043 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.793314 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.793317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.793353 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:48.793460 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.793465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.794689 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.794691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.795704 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.795712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.796458 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.796462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.796539 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.796588 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.796679 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.796757 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":44,"thread_start_us":78,"threads_started":1}
I20260619 16:46:48.797086 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.792582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:53501 (local address 127.3.233.130:41143)
0619 16:46:48.792736 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.792740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.792918 (+ 178us) server_negotiation.cc:407] Connection header received
0619 16:46:48.793335 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.793338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.793426 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.793526 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:46:48.793822 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.794367 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.795935 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.796384 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.796540 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.796605 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.796848 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.796932 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:46:48.797183 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.792488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:41143 (local address 127.3.233.129:53501)
0619 16:46:48.792791 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:48.792808 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:48.792961 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:48.793536 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:48.793539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:48.793556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:46:48.793690 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.793697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.794525 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.794528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.795786 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:48.795796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.796537 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.796541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:48.796718 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.796737 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:46:48.796833 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:46:48.796906 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":49,"thread_start_us":81,"threads_started":1}
I20260619 16:46:48.797546 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60501da624ab4e3fa1cafbbdd1675de5" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18" is_pre_election: true
I20260619 16:46:48.797657 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60501da624ab4e3fa1cafbbdd1675de5" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf03660bbd040468d8fba78d431ef44" is_pre_election: true
I20260619 16:46:48.797741 11957 raft_consensus.cc:2454] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 0.
I20260619 16:46:48.797837 11817 raft_consensus.cc:2454] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 0.
I20260619 16:46:48.798220 11609 leader_election.cc:302] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00e37db1a6344da68217d7d7374c8503, 8bf03660bbd040468d8fba78d431ef44, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:48.798467 12037 raft_consensus.cc:2781] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:48.798538 12037 raft_consensus.cc:482] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:48.798593 12037 raft_consensus.cc:3037] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.799678 12037 raft_consensus.cc:504] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.800092 12037 leader_election.cc:288] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 election: Requested vote from peers 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143), b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937)
I20260619 16:46:48.800432 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60501da624ab4e3fa1cafbbdd1675de5" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf03660bbd040468d8fba78d431ef44"
I20260619 16:46:48.800457 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60501da624ab4e3fa1cafbbdd1675de5" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18"
I20260619 16:46:48.800537 11817 raft_consensus.cc:3037] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.800616 11957 raft_consensus.cc:3037] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:48.801496 11957 raft_consensus.cc:2454] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 1.
I20260619 16:46:48.801509 11817 raft_consensus.cc:2454] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 1.
I20260619 16:46:48.801774 11608 leader_election.cc:302] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e37db1a6344da68217d7d7374c8503, 8bf03660bbd040468d8fba78d431ef44; no voters:
I20260619 16:46:48.801930 12037 raft_consensus.cc:2781] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:48.802135 12037 raft_consensus.cc:686] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 1 LEADER]: Becoming Leader. State: Replica: 00e37db1a6344da68217d7d7374c8503, State: Running, Role: LEADER
I20260619 16:46:48.802397 12037 consensus_queue.cc:237] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.804728 11518 catalog_manager.cc:5495] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00e37db1a6344da68217d7d7374c8503 (127.3.233.129). New cstate: current_term: 1 leader_uuid: "00e37db1a6344da68217d7d7374c8503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:48.822715 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:48.819009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.3.233.129:34557)
0619 16:46:48.819318 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:48.819325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:48.819336 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:46:48.819403 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:48.819405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:48.819456 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:48.819591 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:48.819841 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.820374 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.821222 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:48.821469 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:48.821586 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:48.821853 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:48.822298 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:48.822426 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:48.822484 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 16:46:48.822539 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
I20260619 16:46:48.825503 11957 raft_consensus.cc:1261] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Refusing update from remote peer 00e37db1a6344da68217d7d7374c8503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:48.825503 11817 raft_consensus.cc:1261] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Refusing update from remote peer 00e37db1a6344da68217d7d7374c8503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:48.826030 12037 consensus_queue.cc:1035] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:48.826244 12119 consensus_queue.cc:1035] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:48.837512 11515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:48.837653 11515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:48.839725 11515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Sending DeleteTablet for 3 replicas of tablet 60501da624ab4e3fa1cafbbdd1675de5
I20260619 16:46:48.840435 11937 tablet_service.cc:1514] Processing DeleteTablet for tablet 60501da624ab4e3fa1cafbbdd1675de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:48 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260619 16:46:48.840435 11657 tablet_service.cc:1514] Processing DeleteTablet for tablet 60501da624ab4e3fa1cafbbdd1675de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:48 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:48596
I20260619 16:46:48.840505 11797 tablet_service.cc:1514] Processing DeleteTablet for tablet 60501da624ab4e3fa1cafbbdd1675de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:48 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260619 16:46:48.840642 12100 tablet_replica.cc:330] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: stopping tablet replica
I20260619 16:46:48.840706 12099 tablet_replica.cc:330] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: stopping tablet replica
I20260619 16:46:48.840728 12101 tablet_replica.cc:330] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: stopping tablet replica
I20260619 16:46:48.840768 12100 raft_consensus.cc:2229] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:48.840799 12099 raft_consensus.cc:2229] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:46:48.840849 12101 raft_consensus.cc:2229] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:48.840898 12100 raft_consensus.cc:2258] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:48.840958 12101 raft_consensus.cc:2258] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:48.841089 12099 raft_consensus.cc:2258] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:48.841681 12100 ts_tablet_manager.cc:1905] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:48.841683 12101 ts_tablet_manager.cc:1905] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:48.842155 11515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:46:48.842557 11515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:46:48.844187 12100 ts_tablet_manager.cc:1918] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:48.844261 12100 log.cc:1196] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/60501da624ab4e3fa1cafbbdd1675de5
I20260619 16:46:48.844280 12101 ts_tablet_manager.cc:1918] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:48.844350 12101 log.cc:1196] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/60501da624ab4e3fa1cafbbdd1675de5
I20260619 16:46:48.844534 12100 ts_tablet_manager.cc:1939] T 60501da624ab4e3fa1cafbbdd1675de5 P b1e23afbdf574930ac825d79e899ed18: Deleting consensus metadata
I20260619 16:46:48.844590 12101 ts_tablet_manager.cc:1939] T 60501da624ab4e3fa1cafbbdd1675de5 P 8bf03660bbd040468d8fba78d431ef44: Deleting consensus metadata
I20260619 16:46:48.845191 11504 catalog_manager.cc:4841] TS 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143): tablet 60501da624ab4e3fa1cafbbdd1675de5 (table test-table [id=98e4f64ce7c441688bd175b9ed566eea]) successfully deleted
I20260619 16:46:48.845548 11505 catalog_manager.cc:4841] TS b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937): tablet 60501da624ab4e3fa1cafbbdd1675de5 (table test-table [id=98e4f64ce7c441688bd175b9ed566eea]) successfully deleted
I20260619 16:46:48.846153 12099 ts_tablet_manager.cc:1905] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:48.848618 12099 ts_tablet_manager.cc:1918] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:46:48.848696 12099 log.cc:1196] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/60501da624ab4e3fa1cafbbdd1675de5
I20260619 16:46:48.848942 12099 ts_tablet_manager.cc:1939] T 60501da624ab4e3fa1cafbbdd1675de5 P 00e37db1a6344da68217d7d7374c8503: Deleting consensus metadata
I20260619 16:46:48.849633 11504 catalog_manager.cc:4841] TS 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557): tablet 60501da624ab4e3fa1cafbbdd1675de5 (table test-table [id=98e4f64ce7c441688bd175b9ed566eea]) successfully deleted
I20260619 16:46:48.850716 11937 tablet_service.cc:1467] Processing CreateTablet for tablet 44bda9f33c0848429f661c70024ab77d (DEFAULT_TABLE table=test-table [id=5e3fb0b4c74249878a996e00d5e970dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:48.850747 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 44bda9f33c0848429f661c70024ab77d (DEFAULT_TABLE table=test-table [id=5e3fb0b4c74249878a996e00d5e970dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:48.851018 11937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bda9f33c0848429f661c70024ab77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.851080 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bda9f33c0848429f661c70024ab77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.852771 12031 tablet_bootstrap.cc:492] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503: Bootstrap starting.
I20260619 16:46:48.853200 12032 tablet_bootstrap.cc:492] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: Bootstrap starting.
I20260619 16:46:48.853575 12031 tablet_bootstrap.cc:654] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.853929 12032 tablet_bootstrap.cc:654] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.854558 11797 tablet_service.cc:1467] Processing CreateTablet for tablet 44bda9f33c0848429f661c70024ab77d (DEFAULT_TABLE table=test-table [id=5e3fb0b4c74249878a996e00d5e970dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:46:48.854813 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44bda9f33c0848429f661c70024ab77d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:48.854856 12031 tablet_bootstrap.cc:492] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503: No bootstrap required, opened a new log
I20260619 16:46:48.854914 12031 ts_tablet_manager.cc:1397] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:46:48.855319 12031 raft_consensus.cc:348] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.855399 12031 raft_consensus.cc:374] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.855420 12031 raft_consensus.cc:729] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e37db1a6344da68217d7d7374c8503, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.855548 12031 consensus_queue.cc:260] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.855808 12031 ts_tablet_manager.cc:1428] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:48.857297 12033 tablet_bootstrap.cc:492] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: Bootstrap starting.
I20260619 16:46:48.858081 12033 tablet_bootstrap.cc:654] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:48.859179 12032 tablet_bootstrap.cc:492] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: No bootstrap required, opened a new log
I20260619 16:46:48.859269 12032 ts_tablet_manager.cc:1397] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260619 16:46:48.859658 12032 raft_consensus.cc:348] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.859773 12032 raft_consensus.cc:374] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.859809 12032 raft_consensus.cc:729] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e23afbdf574930ac825d79e899ed18, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.859927 12032 consensus_queue.cc:260] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.860191 12032 ts_tablet_manager.cc:1428] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:46:48.860666 12033 tablet_bootstrap.cc:492] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: No bootstrap required, opened a new log
I20260619 16:46:48.860749 12033 ts_tablet_manager.cc:1397] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 16:46:48.861123 12033 raft_consensus.cc:348] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.861235 12033 raft_consensus.cc:374] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:48.861265 12033 raft_consensus.cc:729] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf03660bbd040468d8fba78d431ef44, State: Initialized, Role: FOLLOWER
I20260619 16:46:48.861377 12033 consensus_queue.cc:260] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:48.861616 12033 ts_tablet_manager.cc:1428] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 16:46:49.035267 12037 raft_consensus.cc:482] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:46:49.035408 12037 raft_consensus.cc:504] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:49.035868 12037 leader_election.cc:288] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143), b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937)
I20260619 16:46:49.036404 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bda9f33c0848429f661c70024ab77d" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf03660bbd040468d8fba78d431ef44" is_pre_election: true
I20260619 16:46:49.036404 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bda9f33c0848429f661c70024ab77d" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18" is_pre_election: true
I20260619 16:46:49.036574 11817 raft_consensus.cc:2454] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 0.
I20260619 16:46:49.036583 11957 raft_consensus.cc:2454] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 0.
I20260619 16:46:49.036952 11608 leader_election.cc:302] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00e37db1a6344da68217d7d7374c8503, 8bf03660bbd040468d8fba78d431ef44, b1e23afbdf574930ac825d79e899ed18; no voters:
I20260619 16:46:49.037231 12037 raft_consensus.cc:2781] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:46:49.037312 12037 raft_consensus.cc:482] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:46:49.037365 12037 raft_consensus.cc:3037] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:49.038452 12037 raft_consensus.cc:504] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:49.038810 12037 leader_election.cc:288] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 election: Requested vote from peers 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143), b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937)
I20260619 16:46:49.039220 11817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bda9f33c0848429f661c70024ab77d" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf03660bbd040468d8fba78d431ef44"
I20260619 16:46:49.039247 11957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bda9f33c0848429f661c70024ab77d" candidate_uuid: "00e37db1a6344da68217d7d7374c8503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e23afbdf574930ac825d79e899ed18"
I20260619 16:46:49.039343 11817 raft_consensus.cc:3037] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:49.039386 11957 raft_consensus.cc:3037] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:49.040239 11817 raft_consensus.cc:2454] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 1.
I20260619 16:46:49.040470 11957 raft_consensus.cc:2454] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e37db1a6344da68217d7d7374c8503 in term 1.
I20260619 16:46:49.040539 11608 leader_election.cc:302] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e37db1a6344da68217d7d7374c8503, 8bf03660bbd040468d8fba78d431ef44; no voters:
I20260619 16:46:49.040707 12037 raft_consensus.cc:2781] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:49.040818 12037 raft_consensus.cc:686] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [term 1 LEADER]: Becoming Leader. State: Replica: 00e37db1a6344da68217d7d7374c8503, State: Running, Role: LEADER
I20260619 16:46:49.040977 12037 consensus_queue.cc:237] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } }
I20260619 16:46:49.042984 11515 catalog_manager.cc:5495] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00e37db1a6344da68217d7d7374c8503 (127.3.233.129). New cstate: current_term: 1 leader_uuid: "00e37db1a6344da68217d7d7374c8503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e37db1a6344da68217d7d7374c8503" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 } health_report { overall_health: UNKNOWN } } }
I20260619 16:46:49.106817 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:49.102818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:41143 (local address 127.3.233.129:45921)
0619 16:46:49.103356 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:49.103369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:49.103469 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:49.103743 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:49.103746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:49.103754 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:49.103855 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:49.103860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.104722 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.104725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:49.105452 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:49.105462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.105999 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.106002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:49.106131 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:49.106185 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:49.106502 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:49.106511 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:46:49.106598 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:46:49.106682 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":154,"threads_started":1}
I20260619 16:46:49.106812 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:49.103059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:45921 (local address 127.3.233.130:41143)
0619 16:46:49.103214 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:49.103218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:49.103466 (+ 248us) server_negotiation.cc:407] Connection header received
0619 16:46:49.103560 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:49.103563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:49.103614 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:49.103695 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:49.103966 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.104585 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.105648 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.105878 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.105995 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:49.106346 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:49.106418 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:49.106498 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:49.106611 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 16:46:49.106664 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:46:49.112475 11957 raft_consensus.cc:1261] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Refusing update from remote peer 00e37db1a6344da68217d7d7374c8503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:49.112524 11817 raft_consensus.cc:1261] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Refusing update from remote peer 00e37db1a6344da68217d7d7374c8503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:46:49.112943 12119 consensus_queue.cc:1035] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e23afbdf574930ac825d79e899ed18" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:49.113121 12037 consensus_queue.cc:1035] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf03660bbd040468d8fba78d431ef44" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:46:49.130021 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:49.126589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:47261 (local address 127.3.233.129:34557)
0619 16:46:49.126740 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:49.126747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:49.126814 (+ 67us) server_negotiation.cc:407] Connection header received
0619 16:46:49.126966 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:49.126969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:49.127045 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:49.127119 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:46:49.127386 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.127871 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.128873 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.129136 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.129268 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:49.129537 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:46:49.129622 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:46:49.129712 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:49.129810 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 16:46:49.129863 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:46:49.130031 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:49.126507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34557 (local address 127.3.233.130:47261)
0619 16:46:49.126731 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:49.126744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:49.126855 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:49.127173 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:49.127177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:49.127186 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:46:49.127302 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:49.127307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.128026 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.128029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:49.128728 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:49.128740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:49.129267 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:49.129271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:49.129369 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:49.129408 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:46:49.129718 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:46:49.129727 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:46:49.129800 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:46:49.129890 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260619 16:46:49.149780 11520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:49.149950 11520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:46:49.151978 11520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20efe0fd88104e47903095aed754ebc4: Sending DeleteTablet for 3 replicas of tablet 44bda9f33c0848429f661c70024ab77d
I20260619 16:46:49.152640 11937 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bda9f33c0848429f661c70024ab77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:49 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56150
I20260619 16:46:49.152621 11797 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bda9f33c0848429f661c70024ab77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:49 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260619 16:46:49.152760 11657 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bda9f33c0848429f661c70024ab77d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:46:49 UTC) from {username='oryx', principal='oryx/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:48596
I20260619 16:46:49.152810 12100 tablet_replica.cc:330] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: stopping tablet replica
I20260619 16:46:49.152935 12100 raft_consensus.cc:2229] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:49.153056 12101 tablet_replica.cc:330] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: stopping tablet replica
I20260619 16:46:49.153127 12100 raft_consensus.cc:2258] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:49.153232 12101 raft_consensus.cc:2229] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:46:49.153270 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 11592
I20260619 16:46:49.153357 12101 raft_consensus.cc:2258] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:46:49.153527 12099 tablet_replica.cc:330] T 44bda9f33c0848429f661c70024ab77d P 00e37db1a6344da68217d7d7374c8503: stopping tablet replica
I20260619 16:46:49.154619 12101 ts_tablet_manager.cc:1905] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:49.155421 12100 ts_tablet_manager.cc:1905] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:46:49.157423 12101 ts_tablet_manager.cc:1918] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:49.157503 12101 log.cc:1196] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/44bda9f33c0848429f661c70024ab77d
I20260619 16:46:49.157759 12101 ts_tablet_manager.cc:1939] T 44bda9f33c0848429f661c70024ab77d P 8bf03660bbd040468d8fba78d431ef44: Deleting consensus metadata
I20260619 16:46:49.158221 12100 ts_tablet_manager.cc:1918] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:46:49.158299 12100 log.cc:1196] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/44bda9f33c0848429f661c70024ab77d
I20260619 16:46:49.158506 11504 catalog_manager.cc:4841] TS 8bf03660bbd040468d8fba78d431ef44 (127.3.233.130:41143): tablet 44bda9f33c0848429f661c70024ab77d (table test-table [id=5e3fb0b4c74249878a996e00d5e970dd]) successfully deleted
I20260619 16:46:49.158565 12100 ts_tablet_manager.cc:1939] T 44bda9f33c0848429f661c70024ab77d P b1e23afbdf574930ac825d79e899ed18: Deleting consensus metadata
I20260619 16:46:49.159209 11505 catalog_manager.cc:4841] TS b1e23afbdf574930ac825d79e899ed18 (127.3.233.131:38937): tablet 44bda9f33c0848429f661c70024ab77d (table test-table [id=5e3fb0b4c74249878a996e00d5e970dd]) successfully deleted
W20260619 16:46:49.161523 11504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:34557 (error 108)
I20260619 16:46:49.161651 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 11732
I20260619 16:46:49.163581 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:49.163180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34557 (local address 127.0.0.1:48602)
0619 16:46:49.163417 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:49.163494 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:34557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
W20260619 16:46:49.163722 11504 catalog_manager.cc:4584] TS 00e37db1a6344da68217d7d7374c8503 (127.3.233.129:34557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44bda9f33c0848429f661c70024ab77d: Network error: Client connection negotiation failed: client connection to 127.3.233.129:34557: connect: Connection refused (error 111)
I20260619 16:46:49.169369 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 11872
I20260619 16:46:49.176316 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 11488
2026-06-19T16:46:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3542 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:49 dist-test-slave-cgwx krb5kdc[12144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:49 dist-test-slave-cgwx krb5kdc[12144](info): set up 2 sockets
Jun 19 16:46:49 dist-test-slave-cgwx krb5kdc[12144](info): commencing operation
krb5kdc: starting...
W20260619 16:46:50.227949 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:50 dist-test-slave-cgwx krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:50.346098 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37235
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:37235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:50.456764 12157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:50.457068 12157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:50.457113 12157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:50.461277 12157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:50.461335 12157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:50.461356 12157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:50.461373 12157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:50.461388 12157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:50.466785 12157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:37235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:50.467940 12157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:50.469089 12157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:50.477805 12162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:50.477823 12163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:50.478170 12157 server_base.cc:1034] running on GCE node
W20260619 16:46:50.477820 12165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:50.479079 12157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:50.480232 12157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:50.481418 12157 hybrid_clock.cc:648] HybridClock initialized: now 1781887610481406 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T16:46:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1316 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:46:50 dist-test-slave-cgwx krb5kdc[12170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:46:50 dist-test-slave-cgwx krb5kdc[12170](info): set up 2 sockets
Jun 19 16:46:50 dist-test-slave-cgwx krb5kdc[12170](info): commencing operation
krb5kdc: starting...
W20260619 16:46:51.563865 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:46:51 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:46:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:46:51.686736 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38471
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:38471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:51.802412 12183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:51.802738 12183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:51.802819 12183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:51.806841 12183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:51.806965 12183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:51.807013 12183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:51.807078 12183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:51.807118 12183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:51.812429 12183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:38471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:51.813606 12183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:51.814621 12183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:51.822741 12189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:51.822731 12191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:51.822737 12188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:51.823123 12183 server_base.cc:1034] running on GCE node
I20260619 16:46:51.823715 12183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:51.824869 12183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:51.826069 12183 hybrid_clock.cc:648] HybridClock initialized: now 1781887611826054 us; error 36 us; skew 500 ppm
Jun 19 16:46:51 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887611, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:51.829288 12183 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:51.830248 12183 webserver.cc:460] Webserver started at http://127.3.233.190:36585/ using document root <none> and password file <none>
I20260619 16:46:51.830826 12183 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:51.830917 12183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:51.831166 12183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:51.833138 12183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "041241c041664ade99f576bf92537931"
format_stamp: "Formatted at 2026-06-19 16:46:51 on dist-test-slave-cgwx"
server_key: "f6d7561ab0013ae57b6224dab67afac0"
server_key_iv: "0a2e7952871211f67d4bf9db75d4b262"
server_key_version: "encryptionkey@0"
I20260619 16:46:51.833647 12183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "041241c041664ade99f576bf92537931"
format_stamp: "Formatted at 2026-06-19 16:46:51 on dist-test-slave-cgwx"
server_key: "f6d7561ab0013ae57b6224dab67afac0"
server_key_iv: "0a2e7952871211f67d4bf9db75d4b262"
server_key_version: "encryptionkey@0"
I20260619 16:46:51.837386 12183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 16:46:51.839810 12198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:51.841189 12183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:46:51.841338 12183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "041241c041664ade99f576bf92537931"
format_stamp: "Formatted at 2026-06-19 16:46:51 on dist-test-slave-cgwx"
server_key: "f6d7561ab0013ae57b6224dab67afac0"
server_key_iv: "0a2e7952871211f67d4bf9db75d4b262"
server_key_version: "encryptionkey@0"
I20260619 16:46:51.841461 12183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:51.854609 12183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:51.857723 12183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:51.857939 12183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:51.866122 12183 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:38471
I20260619 16:46:51.866148 12250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:38471 every 8 connection(s)
I20260619 16:46:51.867281 12183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:51.870043 12251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:46:51.873334 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 12183
I20260619 16:46:51.873458 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:46:51.873782 4006 external_mini_cluster.cc:1428] Setting key dcfd7c309a2b10cf51480ef09c50d0ea
I20260619 16:46:51.876242 12251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Bootstrap starting.
I20260619 16:46:51.878753 12251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Neither blocks nor log segments found. Creating new log.
I20260619 16:46:51.879539 12251 log.cc:824] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:51.881750 12251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: No bootstrap required, opened a new log
Jun 19 16:46:51 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:51.886449 12251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:51.886760 12251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:46:51.886821 12251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041241c041664ade99f576bf92537931, State: Initialized, Role: FOLLOWER
I20260619 16:46:51.887348 12251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:51.887511 12251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:51.887576 12251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:51.887720 12251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:46:51.888928 12251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:51.889328 12251 leader_election.cc:302] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041241c041664ade99f576bf92537931; no voters:
I20260619 16:46:51.889688 12251 leader_election.cc:288] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:46:51.890072 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:51.875325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.3.233.190:38471)
0619 16:46:51.875772 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:51.875795 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:51.875832 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:46:51.876593 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:51.876613 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:51.876967 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:51.877284 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:51.878227 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:51.879506 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:51.880563 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:51.881156 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:51.881290 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:51.884346 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:51.884354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:51.884379 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:51.884417 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:51.886489 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:51.887164 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:51.887170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:51.887176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:51.887313 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:51.887632 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:51.887638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:51.887639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:51.888188 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:51.888381 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:51.888693 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 16:46:51.889010 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":171,"threads_started":1}
I20260619 16:46:51.890131 12257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:46:51.890648 12257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 LEADER]: Becoming Leader. State: Replica: 041241c041664ade99f576bf92537931, State: Running, Role: LEADER
I20260619 16:46:51.890990 12251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:51.891104 12257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:51.893925 12258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041241c041664ade99f576bf92537931. Latest consensus state: current_term: 1 leader_uuid: "041241c041664ade99f576bf92537931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } } }
I20260619 16:46:51.893977 12256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "041241c041664ade99f576bf92537931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } } }
I20260619 16:46:51.894066 12258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:51.894069 12256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: This master's current role is: LEADER
W20260619 16:46:51.896421 12270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:46:51.896525 12270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:46:51.896610 12271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:51.897617 12271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:51.903606 12271 catalog_manager.cc:1347] Generated new cluster ID: 4f2a446494454b9398502927fb33847d
I20260619 16:46:51.903682 12271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:51.909900 12271 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:46:51.911128 12271 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:51.916643 12271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Generated new TSK 0
I20260619 16:46:51.917413 12271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:46:51.951786 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38471
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:52.063704 12277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:52.063985 12277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:52.064031 12277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:52.068238 12277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:52.068313 12277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:52.068403 12277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:52.074126 12277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:52.075398 12277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:52.076465 12277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:52.086315 12283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:52.086320 12285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:52.086320 12282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:52.086722 12277 server_base.cc:1034] running on GCE node
I20260619 16:46:52.087280 12277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:52.087993 12277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:52.089190 12277 hybrid_clock.cc:648] HybridClock initialized: now 1781887612089162 us; error 36 us; skew 500 ppm
Jun 19 16:46:52 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:52.092235 12277 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:46:52.093214 12277 webserver.cc:460] Webserver started at http://127.3.233.129:33035/ using document root <none> and password file <none>
I20260619 16:46:52.093780 12277 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:52.093840 12277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:52.094070 12277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:52.096190 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "62adad29270c4208a369a79aeae6b598"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "7fcf0c4156dc22c9e7ab1ace2c3682f4"
server_key_iv: "a305dcfb48488a45e782edb8ead407de"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.096748 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "62adad29270c4208a369a79aeae6b598"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "7fcf0c4156dc22c9e7ab1ace2c3682f4"
server_key_iv: "a305dcfb48488a45e782edb8ead407de"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.100590 12277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:52.103209 12292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.104465 12277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:52.104604 12277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "62adad29270c4208a369a79aeae6b598"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "7fcf0c4156dc22c9e7ab1ace2c3682f4"
server_key_iv: "a305dcfb48488a45e782edb8ead407de"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.104727 12277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:52.119592 12277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:52.122605 12277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:52.122825 12277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:52.123505 12277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:52.124351 12277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:52.124423 12277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.124488 12277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:52.124532 12277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.133383 12277 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:41309
I20260619 16:46:52.133409 12405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:41309 every 8 connection(s)
I20260619 16:46:52.134279 12277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:52.138095 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 12277
I20260619 16:46:52.138231 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:46:52.138561 4006 external_mini_cluster.cc:1428] Setting key 55e5266b7cf608e3cd8130e4061ca8de
Jun 19 16:46:52 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:46:52.149775 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:52.136119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:33701 (local address 127.3.233.190:38471)
0619 16:46:52.136305 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:52.136310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:52.137375 (+ 1065us) server_negotiation.cc:407] Connection header received
0619 16:46:52.138276 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:52.138282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:52.138339 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:52.138443 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:52.140026 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.140803 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.142346 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.142575 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.142658 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:52.145708 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:52.145714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:52.145721 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:52.145745 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:52.147463 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:52.148055 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:52.148058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:52.148059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:52.148137 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:52.148509 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:52.148513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:52.148515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:52.148781 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:52.148894 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:52.149422 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 16:46:52.149594 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:46:52.150708 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:52.136518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:33701)
0619 16:46:52.137224 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:52.137268 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:52.138030 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:52.138594 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:52.138600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:52.139149 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:52.139829 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:52.139850 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.140959 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.140961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:52.142218 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:52.142232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.142732 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.142739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:52.143022 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:52.143682 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:52.143707 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:52.145529 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:52.147591 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:52.147598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:52.147616 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:52.147888 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:52.148239 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:52.148243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:52.148245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:52.148390 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:52.148902 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:52.148912 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:52.149171 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 16:46:52.149369 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 16:46:52.149724 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:46:52.152254 12406 heartbeater.cc:347] Connected to a master server at 127.3.233.190:38471
I20260619 16:46:52.152509 12406 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:52.153123 12406 heartbeater.cc:511] Master 127.3.233.190:38471 requested a full tablet report, sending...
I20260619 16:46:52.154753 12213 ts_manager.cc:194] Registered new tserver with Master: 62adad29270c4208a369a79aeae6b598 (127.3.233.129:41309)
I20260619 16:46:52.155933 12213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:33701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:46:52.167474 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38471
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:52.281603 12411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:52.281889 12411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:52.281939 12411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:52.286725 12411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:52.286800 12411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:52.286862 12411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:52.292681 12411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:52.293928 12411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:52.294896 12411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:52.304710 12419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:52.304878 12411 server_base.cc:1034] running on GCE node
W20260619 16:46:52.304807 12416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:52.304837 12417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:52.305428 12411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:52.306185 12411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:52.307379 12411 hybrid_clock.cc:648] HybridClock initialized: now 1781887612307363 us; error 33 us; skew 500 ppm
Jun 19 16:46:52 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:52.310268 12411 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:46:52.311204 12411 webserver.cc:460] Webserver started at http://127.3.233.130:39899/ using document root <none> and password file <none>
I20260619 16:46:52.311800 12411 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:52.311896 12411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:52.312130 12411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:52.314254 12411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "e93175568bb74896a5194582d3ac94db"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "0298dff8fc9ffe4d486b8588eb4aa8b9"
server_key_iv: "b7a84fbe8f09123cdf4700a1ca6e95f6"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.314888 12411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "e93175568bb74896a5194582d3ac94db"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "0298dff8fc9ffe4d486b8588eb4aa8b9"
server_key_iv: "b7a84fbe8f09123cdf4700a1ca6e95f6"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.318868 12411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:52.321521 12426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.322779 12411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:52.322986 12411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "e93175568bb74896a5194582d3ac94db"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "0298dff8fc9ffe4d486b8588eb4aa8b9"
server_key_iv: "b7a84fbe8f09123cdf4700a1ca6e95f6"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.323123 12411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:52.339933 12411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:52.343194 12411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:52.343393 12411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:52.343994 12411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:52.344830 12411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:52.344909 12411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.345000 12411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:52.345052 12411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.354368 12411 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:40047
I20260619 16:46:52.354413 12539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:40047 every 8 connection(s)
I20260619 16:46:52.355373 12411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:52.364503 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 12411
I20260619 16:46:52.364660 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:46:52.364945 4006 external_mini_cluster.cc:1428] Setting key 28b2f5d2d6b5d4676241afa2c1608293
Jun 19 16:46:52 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:52.369642 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:52.357263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:52383 (local address 127.3.233.190:38471)
0619 16:46:52.357425 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:52.357429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:52.358737 (+ 1308us) server_negotiation.cc:407] Connection header received
0619 16:46:52.359709 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:52.359712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:52.359761 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:52.359840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:52.361263 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.361783 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.362979 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.363214 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.363303 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:52.366025 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:52.366029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:52.366032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:52.366064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:52.367570 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:52.368104 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:52.368109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:52.368111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:52.368195 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:52.368541 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:52.368546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:52.368548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:52.368723 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:52.368827 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:52.369340 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 16:46:52.369464 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:46:52.370596 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:52.357651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:52383)
0619 16:46:52.358571 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:52.358617 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:52.359506 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:52.360085 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:52.360093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:52.360460 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:52.361087 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:52.361103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.361916 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.361919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:52.362810 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:52.362819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.363383 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.363386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:52.363547 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:52.364126 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:52.364142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:52.365867 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:52.367693 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:52.367700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:52.367716 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:52.367981 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:52.368287 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:52.368291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:52.368293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:52.368441 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:52.368826 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:52.368832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:52.369106 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 16:46:52.369302 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 16:46:52.369642 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":531,"threads_started":1}
I20260619 16:46:52.372128 12540 heartbeater.cc:347] Connected to a master server at 127.3.233.190:38471
I20260619 16:46:52.372400 12540 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:52.372928 12540 heartbeater.cc:511] Master 127.3.233.190:38471 requested a full tablet report, sending...
I20260619 16:46:52.374092 12213 ts_manager.cc:194] Registered new tserver with Master: e93175568bb74896a5194582d3ac94db (127.3.233.130:40047)
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:46:52.374850 12213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:52383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:46:52.393932 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38471
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:52.507367 12545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:52.507676 12545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:52.507720 12545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:52.512010 12545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:52.512081 12545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:52.512177 12545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:52.517599 12545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:52.518708 12545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:52.519804 12545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:46:52.529834 12545 server_base.cc:1034] running on GCE node
W20260619 16:46:52.529774 12551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:52.529839 12550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:52.529774 12553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:52.530408 12545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:52.531163 12545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:52.532338 12545 hybrid_clock.cc:648] HybridClock initialized: now 1781887612532286 us; error 64 us; skew 500 ppm
Jun 19 16:46:52 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:52.535331 12545 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:46:52.536217 12545 webserver.cc:460] Webserver started at http://127.3.233.131:33315/ using document root <none> and password file <none>
I20260619 16:46:52.536777 12545 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:52.536866 12545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:52.537114 12545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:46:52.539186 12545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "52c5660e11794e46887b139cb3e7c39e"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "70d0404ad9e3520193ae072b869d756e"
server_key_iv: "a567e3978d4be340b83b9d452261519f"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.539752 12545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "52c5660e11794e46887b139cb3e7c39e"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "70d0404ad9e3520193ae072b869d756e"
server_key_iv: "a567e3978d4be340b83b9d452261519f"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.543542 12545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:46:52.546056 12560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.547309 12545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:46:52.547439 12545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "52c5660e11794e46887b139cb3e7c39e"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "70d0404ad9e3520193ae072b869d756e"
server_key_iv: "a567e3978d4be340b83b9d452261519f"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.547549 12545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:52.565802 12545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:52.568950 12545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:52.569171 12545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:52.569772 12545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:52.570616 12545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:52.570686 12545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.570761 12545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:52.570803 12545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.579844 12545 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:38177
I20260619 16:46:52.579869 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:38177 every 8 connection(s)
I20260619 16:46:52.580711 12545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:52.590538 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 12545
I20260619 16:46:52.590683 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:46:52.591055 4006 external_mini_cluster.cc:1428] Setting key 5afa6a60f3c9782bb9842d01acb75f44
Jun 19 16:46:52 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:46:52.595646 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:52.582703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:43249 (local address 127.3.233.190:38471)
0619 16:46:52.583006 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:52.583011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:52.584518 (+ 1507us) server_negotiation.cc:407] Connection header received
0619 16:46:52.585752 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:52.585755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:52.585801 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:52.585878 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:52.587043 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.587571 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.588684 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.588891 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.588973 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:52.591993 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:52.591997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:52.592000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:52.592031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:52.593655 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:52.594178 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:52.594181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:52.594182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:52.594253 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:52.594544 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:52.594546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:52.594547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:52.594732 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:52.594805 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:52.595405 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 16:46:52.595500 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260619 16:46:52.596423 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:52.583270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:43249)
0619 16:46:52.584357 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:52.584400 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:52.585549 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:52.586037 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:52.586045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:52.586391 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:52.586886 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:52.586898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.587686 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.587689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:52.588579 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:52.588588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:52.589051 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:52.589054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:52.589205 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:52.589883 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:52.589899 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:52.591819 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:46:52.593753 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:52.593758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:52.593770 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:52.594084 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:52.594326 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:46:52.594328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:46:52.594329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:46:52.594458 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:46:52.594822 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:46:52.594827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:46:52.595181 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 16:46:52.595336 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 16:46:52.595598 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":169,"threads_started":1}
I20260619 16:46:52.597515 12674 heartbeater.cc:347] Connected to a master server at 127.3.233.190:38471
I20260619 16:46:52.597743 12674 heartbeater.cc:464] Registering TS with master...
I20260619 16:46:52.598273 12674 heartbeater.cc:511] Master 127.3.233.190:38471 requested a full tablet report, sending...
I20260619 16:46:52.599397 12213 ts_manager.cc:194] Registered new tserver with Master: 52c5660e11794e46887b139cb3e7c39e (127.3.233.131:38177)
I20260619 16:46:52.599985 12213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:43249
I20260619 16:46:52.605504 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab.
Entry for principal oryx/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab.
WARNING: no policy specified for oryx/127.3.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab.
Entry for principal oryx/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab.
WARNING: no policy specified for oryx/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab.
Entry for principal oryx/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab.
I20260619 16:46:52.681962 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 12277
I20260619 16:46:52.687882 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 12411
I20260619 16:46:52.693368 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 12545
I20260619 16:46:52.699312 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 12183
I20260619 16:46:52.707460 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38471
--webserver_interface=127.3.233.190
--webserver_port=36585
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:38471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:46:52.817060 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:52.817385 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:52.817466 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:52.821481 12683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:46:52.821579 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:52.821616 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:52.821681 12683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:46:52.821719 12683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:46:52.827157 12683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:38471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=36585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:52.828302 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:52.829344 12683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:52.838096 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:52.838153 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:52.838338 12683 server_base.cc:1034] running on GCE node
W20260619 16:46:52.838565 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:52.839195 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:52.840350 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:52.841559 12683 hybrid_clock.cc:648] HybridClock initialized: now 1781887612841529 us; error 47 us; skew 500 ppm
Jun 19 16:46:52 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:52.844472 12683 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:46:52.845424 12683 webserver.cc:460] Webserver started at http://127.3.233.190:36585/ using document root <none> and password file <none>
I20260619 16:46:52.845991 12683 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:52.846159 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:52.850085 12683 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:46:52.852576 12698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:52.853842 12683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:46:52.853979 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "041241c041664ade99f576bf92537931"
format_stamp: "Formatted at 2026-06-19 16:46:51 on dist-test-slave-cgwx"
server_key: "f6d7561ab0013ae57b6224dab67afac0"
server_key_iv: "0a2e7952871211f67d4bf9db75d4b262"
server_key_version: "encryptionkey@0"
I20260619 16:46:52.854537 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:52.874749 12683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:52.877774 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:52.877990 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:52.885718 12683 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:38471
I20260619 16:46:52.885735 12750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:38471 every 8 connection(s)
I20260619 16:46:52.886904 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:46:52.891396 12751 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:46:52.893072 12751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Bootstrap starting.
I20260619 16:46:52.894032 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 12683
I20260619 16:46:52.894515 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:41309
--local_ip_for_outbound_sockets=127.3.233.129
--tserver_master_addrs=127.3.233.190:38471
--webserver_port=33035
--webserver_interface=127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:46:52.909993 12751 log.cc:824] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Log is configured to *not* fsync() on all Append() calls
I20260619 16:46:52.916688 12751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:46:52.917058 12751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Bootstrap complete.
I20260619 16:46:52.921298 12751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:52.921727 12751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041241c041664ade99f576bf92537931, State: Initialized, Role: FOLLOWER
I20260619 16:46:52.922199 12751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:52.922338 12751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:46:52.922458 12751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:46:52.922605 12751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:46:52.924564 12751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:52.924891 12751 leader_election.cc:302] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041241c041664ade99f576bf92537931; no voters:
I20260619 16:46:52.925192 12751 leader_election.cc:288] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:46:52.925343 12755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:46:52.925583 12755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [term 2 LEADER]: Becoming Leader. State: Replica: 041241c041664ade99f576bf92537931, State: Running, Role: LEADER
I20260619 16:46:52.925961 12755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } }
I20260619 16:46:52.926386 12751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:46:52.927314 12756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "041241c041664ade99f576bf92537931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } } }
I20260619 16:46:52.927431 12756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:52.928094 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041241c041664ade99f576bf92537931. Latest consensus state: current_term: 2 leader_uuid: "041241c041664ade99f576bf92537931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041241c041664ade99f576bf92537931" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38471 } } }
I20260619 16:46:52.928176 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931 [sys.catalog]: This master's current role is: LEADER
I20260619 16:46:52.928586 12764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:46:52.932024 12764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:46:52.932641 12764 catalog_manager.cc:1259] Loaded cluster ID: 4f2a446494454b9398502927fb33847d
I20260619 16:46:52.932699 12764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:46:52.934954 12764 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:46:52.935591 12764 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 041241c041664ade99f576bf92537931: Loaded TSK: 0
I20260619 16:46:52.936782 12764 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:46:53.013715 12753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:53.013985 12753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:53.014029 12753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:53.018311 12753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:53.018386 12753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:53.018448 12753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:46:53.023864 12753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.129.keytab
--principal=oryx/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:41309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=33035
--tserver_master_addrs=127.3.233.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:53.024916 12753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:53.025996 12753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:53.036887 12778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:53.036882 12779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:53.037240 12753 server_base.cc:1034] running on GCE node
W20260619 16:46:53.037390 12781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:53.037837 12753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:53.038619 12753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:53.039800 12753 hybrid_clock.cc:648] HybridClock initialized: now 1781887613039763 us; error 57 us; skew 500 ppm
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887613, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:53.042608 12753 init.cc:377] Logged in from keytab as oryx/127.3.233.129@KRBTEST.COM (short username oryx)
I20260619 16:46:53.043557 12753 webserver.cc:460] Webserver started at http://127.3.233.129:33035/ using document root <none> and password file <none>
I20260619 16:46:53.044116 12753 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:53.044188 12753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:53.048151 12753 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:46:53.050316 12788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.051680 12753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:53.051823 12753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "62adad29270c4208a369a79aeae6b598"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "7fcf0c4156dc22c9e7ab1ace2c3682f4"
server_key_iv: "a305dcfb48488a45e782edb8ead407de"
server_key_version: "encryptionkey@0"
I20260619 16:46:53.052381 12753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:53.065712 12753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:53.069098 12753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:53.069317 12753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:53.069913 12753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:53.070811 12753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:53.070883 12753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.070986 12753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:53.071029 12753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.080005 12753 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:41309
I20260619 16:46:53.080026 12901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:41309 every 8 connection(s)
I20260619 16:46:53.081162 12753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:46:53.091140 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 12753
I20260619 16:46:53.091595 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:40047
--local_ip_for_outbound_sockets=127.3.233.130
--tserver_master_addrs=127.3.233.190:38471
--webserver_port=39899
--webserver_interface=127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:46:53.094868 12904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.083455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:33181 (local address 127.3.233.190:38471)
0619 16:46:53.083989 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.084012 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.084450 (+ 438us) server_negotiation.cc:407] Connection header received
0619 16:46:53.085539 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.085550 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.085873 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.086207 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.087434 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.088410 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.089608 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.090109 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.090302 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.094218 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:33181: BlockingRecv error: recv got EOF from 127.3.233.129:33181 (error 108)
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":131,"threads_started":1}
I20260619 16:46:53.094949 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.083581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:33181)
0619 16:46:53.084281 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.084322 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.085124 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.086293 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.086300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.086755 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.087248 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.087262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.088561 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.088565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.089477 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.089486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.090284 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.090288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.090470 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.091192 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.091210 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.093892 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":123,"threads_started":1}
W20260619 16:46:53.095582 12902 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:38471 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.233.190:38471: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:53.101373 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.096090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:52991)
0619 16:46:53.096204 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.096222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.096305 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.096570 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.096573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.096751 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.096838 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.096843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.097559 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.097565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.098122 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.098129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.098575 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.098578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.098678 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.099245 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.099256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.101226 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 16:46:53.101457 12904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.096166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:52991 (local address 127.3.233.190:38471)
0619 16:46:53.096279 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.096282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.096295 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:53.096416 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.096419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.096465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.096558 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.096947 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.097456 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.098238 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.098470 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.098576 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.101343 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:52991: BlockingRecv error: recv got EOF from 127.3.233.129:52991 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:53.106981 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.102039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:44369)
0619 16:46:53.102126 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.102141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.102241 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.102478 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.102481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.102626 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.102712 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.102717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.103490 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.103496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.104010 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.104017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.104464 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.104467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.104562 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.105053 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.105064 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.106815 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 16:46:53.107163 12902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:46:53.107067 12904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.102136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:44369 (local address 127.3.233.190:38471)
0619 16:46:53.102270 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.102277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.102303 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:46:53.102337 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.102339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.102385 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.102466 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.102821 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.103387 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.104125 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.104357 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.104464 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.106952 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:44369: BlockingRecv error: recv got EOF from 127.3.233.129:44369 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 16:46:53.203866 12906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:53.204140 12906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:53.204183 12906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:53.208280 12906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:53.208344 12906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:53.208438 12906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:46:53.213809 12906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.130.keytab
--principal=oryx/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:40047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=39899
--tserver_master_addrs=127.3.233.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:53.215075 12906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:53.216156 12906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:53.226296 12914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:53.226233 12911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:53.226227 12912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:53.226722 12906 server_base.cc:1034] running on GCE node
I20260619 16:46:53.227265 12906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:53.228263 12906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:53.229588 12906 hybrid_clock.cc:648] HybridClock initialized: now 1781887613229555 us; error 42 us; skew 500 ppm
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887613, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:53.234313 12906 init.cc:377] Logged in from keytab as oryx/127.3.233.130@KRBTEST.COM (short username oryx)
I20260619 16:46:53.235572 12906 webserver.cc:460] Webserver started at http://127.3.233.130:39899/ using document root <none> and password file <none>
I20260619 16:46:53.236306 12906 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:53.236414 12906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:53.242326 12906 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:46:53.245303 12921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.246904 12906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:53.247082 12906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "e93175568bb74896a5194582d3ac94db"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "0298dff8fc9ffe4d486b8588eb4aa8b9"
server_key_iv: "b7a84fbe8f09123cdf4700a1ca6e95f6"
server_key_version: "encryptionkey@0"
I20260619 16:46:53.247833 12906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:53.266959 12906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:53.270867 12906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:53.271123 12906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:53.271744 12906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:53.272588 12906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:53.272668 12906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.272763 12906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:53.272812 12906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.281994 12906 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:40047
I20260619 16:46:53.282045 13034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:40047 every 8 connection(s)
I20260619 16:46:53.283123 12906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:46:53.288676 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 12906
I20260619 16:46:53.289218 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:38177
--local_ip_for_outbound_sockets=127.3.233.131
--tserver_master_addrs=127.3.233.190:38471
--webserver_port=33315
--webserver_interface=127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:46:53.295615 12904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.285267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:49857 (local address 127.3.233.190:38471)
0619 16:46:53.285421 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.285425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.286486 (+ 1061us) server_negotiation.cc:407] Connection header received
0619 16:46:53.287492 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.287495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.287559 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.287668 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.288934 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.289788 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.291462 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.291678 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.291817 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.295441 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:49857: BlockingRecv error: recv got EOF from 127.3.233.130:49857 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:46:53.296103 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.285643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:49857)
0619 16:46:53.286322 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.286366 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.287274 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.287851 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.287858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.288231 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.288752 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.288766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.289940 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.289942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.291331 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.291346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.291866 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.291870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.292022 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.292661 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.292677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.295251 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":123,"threads_started":1}
W20260619 16:46:53.296743 13035 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:38471 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.233.190:38471: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:53.303474 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.297192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:47989)
0619 16:46:53.297435 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.297447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.297512 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.297779 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.297781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.297907 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.297995 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.298000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.299260 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.299266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.299939 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.299975 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.300726 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.300729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.300802 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.301382 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.301394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.303370 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260619 16:46:53.303859 12904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.297315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:47989 (local address 127.3.233.190:38471)
0619 16:46:53.297464 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.297471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.297489 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:53.297595 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.297600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.297658 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.297750 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.298105 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.299044 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.300155 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.300585 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.300709 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.303524 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:47989: BlockingRecv error: recv got EOF from 127.3.233.130:47989 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:53.309736 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.304085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:46791)
0619 16:46:53.304233 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.304249 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.304326 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.304638 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.304641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.304803 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.304893 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.304900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.305663 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.305666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.306542 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.306551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.307046 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.307050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.307137 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.307637 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.307658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.309573 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:46:53.309927 12904 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.304322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:46791 (local address 127.3.233.190:38471)
0619 16:46:53.304425 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.304428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.304439 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:46:53.304475 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.304477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.304515 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.304602 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.305027 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.305559 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.306686 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.306900 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.307041 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.309702 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:46791: BlockingRecv error: recv got EOF from 127.3.233.130:46791 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 16:46:53.309967 13035 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:46:53.456305 13038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:46:53.456794 13038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:46:53.456921 13038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:46:53.467968 13038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:46:53.468091 13038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:46:53.468253 13038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:46:53.482121 13038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/krb5kdc/oryx_127.3.233.131.keytab
--principal=oryx/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:38177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=33315
--tserver_master_addrs=127.3.233.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:46:53.483974 13038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:46:53.485600 13038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:46:53.498383 13043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:53.498383 13046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:46:53.498441 13044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:46:53.498978 13038 server_base.cc:1034] running on GCE node
I20260619 16:46:53.499548 13038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:46:53.500550 13038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:46:53.501782 13038 hybrid_clock.cc:648] HybridClock initialized: now 1781887613501743 us; error 53 us; skew 500 ppm
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887613, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:46:53.505741 13038 init.cc:377] Logged in from keytab as oryx/127.3.233.131@KRBTEST.COM (short username oryx)
I20260619 16:46:53.506770 13038 webserver.cc:460] Webserver started at http://127.3.233.131:33315/ using document root <none> and password file <none>
I20260619 16:46:53.507416 13038 fs_manager.cc:359] Metadata directory not provided
I20260619 16:46:53.507491 13038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:46:53.511693 13038 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:46:53.513985 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.515645 13038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:46:53.515790 13038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "52c5660e11794e46887b139cb3e7c39e"
format_stamp: "Formatted at 2026-06-19 16:46:52 on dist-test-slave-cgwx"
server_key: "70d0404ad9e3520193ae072b869d756e"
server_key_iv: "a567e3978d4be340b83b9d452261519f"
server_key_version: "encryptionkey@0"
I20260619 16:46:53.516374 13038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:46:53.531423 13038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:46:53.535184 13038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:46:53.535396 13038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:46:53.536052 13038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:46:53.536934 13038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:46:53.537005 13038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.537077 13038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:46:53.537127 13038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:46:53.547910 13038 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:38177
I20260619 16:46:53.547986 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:38177 every 8 connection(s)
I20260619 16:46:53.548985 13038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:46:53.549670 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13038
I20260619 16:46:53.561937 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.550380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.3.233.190:38471)
0619 16:46:53.550522 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.550526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.550799 (+ 273us) server_negotiation.cc:407] Connection header received
0619 16:46:53.550995 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.551002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.551047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.551166 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.552026 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.553122 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.554165 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.554662 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.554816 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.556693 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:46:53.556703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:46:53.556727 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:46:53.556768 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:46:53.559635 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:53.560123 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:53.560129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:53.560143 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:53.560272 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:46:53.560563 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:46:53.560566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:46:53.560568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:46:53.561017 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:46:53.561229 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:46:53.561493 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:46:53.561734 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:53.565443 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.552710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:54153)
0619 16:46:53.554382 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.554424 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.555460 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.556335 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.556345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.556788 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.557341 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.557356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.558311 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.558314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.559419 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.559428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.560017 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.560024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.560308 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.560959 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.560987 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.564173 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1496,"thread_start_us":134,"threads_started":1}
W20260619 16:46:53.565987 13167 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:38471 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.233.190:38471: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
W20260619 16:46:53.566254 13170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.552481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:54153 (local address 127.3.233.190:38471)
0619 16:46:53.553422 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.553428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.554518 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 16:46:53.555712 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.555718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.555790 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.555911 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.557471 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.558169 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.559538 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.559831 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.559944 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.566058 (+ 6114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:54153: BlockingRecv error: recv got EOF from 127.3.233.131:54153 (error 108)
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":78,"threads_started":1}
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:53.572953 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.566413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:54787)
0619 16:46:53.566674 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.566690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.566772 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.567078 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.567080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.567220 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.567319 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.567324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.568610 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.568613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.569186 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.569195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.569911 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.569914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.569979 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.570550 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.570561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.572854 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185}
W20260619 16:46:53.573254 13170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.566637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:54787 (local address 127.3.233.190:38471)
0619 16:46:53.566791 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.566799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.566818 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:46:53.566865 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.566871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.566951 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.567061 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.567445 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.568478 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.569316 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.569792 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.569928 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.573085 (+ 3157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:54787: BlockingRecv error: recv got EOF from 127.3.233.131:54787 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:53 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:53.580247 13171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:53.573553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:36693)
0619 16:46:53.573649 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:53.573664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:53.573753 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:53.574079 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:53.574081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:53.574200 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:53.574277 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.574281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.575602 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.575609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.576879 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:53.576893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.577350 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.577356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:53.577490 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.578004 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:53.578026 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:53.580085 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 16:46:53.580300 13170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:53.573633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:36693 (local address 127.3.233.190:38471)
0619 16:46:53.573815 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:53.573821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:53.573839 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:46:53.573888 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:53.573894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:53.573954 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:53.574072 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:53.574389 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.575436 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.577007 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:53.577226 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:53.577305 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:53.580196 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:36693: BlockingRecv error: recv got EOF from 127.3.233.131:36693 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 16:46:53.580463 13167 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:46:54 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:54 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:54.113600 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:54.107907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:48085)
0619 16:46:54.108194 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:54.108208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:54.108312 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:54.108570 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:54.108572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:54.108751 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:54.108869 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:54.108874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.109646 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.109648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:54.110237 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:54.110245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.110657 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.110659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:54.110731 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:54.111307 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:54.111319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:54.113430 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
W20260619 16:46:54.113672 13173 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:54.107987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:48085 (local address 127.3.233.190:38471)
0619 16:46:54.108261 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:54.108264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:54.108280 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:46:54.108406 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:54.108408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:54.108454 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:54.108534 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:54.108982 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.109545 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.110338 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.110558 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.110666 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:54.113550 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:48085: BlockingRecv error: recv got EOF from 127.3.233.129:48085 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":58,"threads_started":1}
Jun 19 16:46:54 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:54 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:54.316697 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:54.310717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:43007)
0619 16:46:54.311019 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:54.311034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:54.311143 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:54.311443 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:54.311446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:54.311670 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:54.311792 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:54.311798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.312731 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.312735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:54.313326 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:54.313334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.313815 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.313818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:54.313917 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:54.314509 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:54.314554 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:54.316550 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
W20260619 16:46:54.316839 13173 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:54.310855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:43007 (local address 127.3.233.190:38471)
0619 16:46:54.311039 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:54.311043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:54.311092 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:46:54.311260 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:54.311262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:54.311312 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:54.311390 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:54.311924 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.312590 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.313465 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.313695 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.313811 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:54.316707 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:43007: BlockingRecv error: recv got EOF from 127.3.233.130:43007 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 16:46:54 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:54 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:54.587067 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:54.581125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:35065)
0619 16:46:54.581390 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:54.581403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:54.581504 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:54.581744 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:54.581747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:54.581919 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:54.582013 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:54.582018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.583022 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.583026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:54.583621 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:54.583631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.584133 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.584137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:54.584208 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:54.584798 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:54.584811 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:54.586875 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
W20260619 16:46:54.587284 13173 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:54.581192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:35065 (local address 127.3.233.190:38471)
0619 16:46:54.581317 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:54.581321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:54.581504 (+ 183us) server_negotiation.cc:407] Connection header received
0619 16:46:54.581571 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:54.581573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:54.581621 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:54.581694 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:54.582298 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.582818 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.583770 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:54.583993 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:54.584085 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:54.587140 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:35065: BlockingRecv error: recv got EOF from 127.3.233.131:35065 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 16:46:55 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:55 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:55.119834 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:55.114422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:46803)
0619 16:46:55.114710 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:55.114724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:55.114822 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:55.115187 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:55.115190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:55.115373 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:55.115470 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:55.115475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.116174 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.116176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:55.116726 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:55.116743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.117259 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.117263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:55.117330 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:55.117897 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:55.117909 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:55.119713 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
W20260619 16:46:55.120075 13177 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:55.114516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:46803 (local address 127.3.233.190:38471)
0619 16:46:55.114778 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:55.114782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:55.114796 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:46:55.114995 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:55.114997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:55.115044 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:55.115138 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:55.115571 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.116084 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.116865 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.117141 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.117236 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:55.119912 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:46803: BlockingRecv error: recv got EOF from 127.3.233.129:46803 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
Jun 19 16:46:55 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:55 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:55.323288 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:55.317504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:36831)
0619 16:46:55.317821 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:55.317835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:55.317972 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:55.318302 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:55.318305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:55.318516 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:55.318613 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:55.318618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.319447 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.319452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:55.320025 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:55.320033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.320557 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.320561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:55.320646 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:55.321169 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:55.321191 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:55.323161 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":87,"threads_started":1}
W20260619 16:46:55.323477 13177 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:55.317614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:36831 (local address 127.3.233.190:38471)
0619 16:46:55.317769 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:55.317774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:55.317916 (+ 142us) server_negotiation.cc:407] Connection header received
0619 16:46:55.318142 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:55.318144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:55.318188 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:55.318282 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:55.318724 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.319341 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.320143 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.320434 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.320540 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:55.323317 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:36831: BlockingRecv error: recv got EOF from 127.3.233.130:36831 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:46:55 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:55 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:55.593837 13179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:55.587997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:42229)
0619 16:46:55.588331 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:55.588348 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:55.588460 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:55.588769 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:55.588772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:55.588972 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:55.589069 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:55.589074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.589820 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.589826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:55.590638 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:55.590652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.591239 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.591243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:55.591348 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:55.591883 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:55.591895 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:55.593701 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
W20260619 16:46:55.594003 13177 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:55.588185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:42229 (local address 127.3.233.190:38471)
0619 16:46:55.588332 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:55.588337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:55.588406 (+ 69us) server_negotiation.cc:407] Connection header received
0619 16:46:55.588597 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:55.588600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:55.588655 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:55.588742 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:55.589184 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.589717 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.590776 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:55.591084 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:55.591190 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:55.593865 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:42229: BlockingRecv error: recv got EOF from 127.3.233.131:42229 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:46:56 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:56 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:56.126845 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:56.120618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:39093)
0619 16:46:56.120918 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:56.120931 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:56.121031 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:56.121425 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:56.121427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:56.121606 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:56.121709 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:56.121715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.122531 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.122534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:56.123275 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:56.123285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.123738 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.123741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:56.123805 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:56.124401 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:56.124413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:56.126711 (+ 2298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
W20260619 16:46:56.127053 13181 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:56.120696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:39093 (local address 127.3.233.190:38471)
0619 16:46:56.120964 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:56.120968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:56.120985 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:46:56.121238 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:56.121241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:56.121289 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:56.121373 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:56.121827 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.122415 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.123395 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.123617 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.123701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:56.126874 (+ 3173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:39093: BlockingRecv error: recv got EOF from 127.3.233.129:39093 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
Jun 19 16:46:56 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:56 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:56.330206 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:56.324021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:48969)
0619 16:46:56.324362 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:56.324377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:56.324514 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:56.324818 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:56.324821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:56.325034 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:56.325125 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:56.325130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.325943 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.325946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:56.326597 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:56.326606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.327189 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.327192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:56.327275 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:56.327880 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:56.327900 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:56.330033 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
W20260619 16:46:56.330452 13181 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:56.324140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:48969 (local address 127.3.233.190:38471)
0619 16:46:56.324314 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:56.324319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:56.324459 (+ 140us) server_negotiation.cc:407] Connection header received
0619 16:46:56.324640 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:56.324642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:56.324708 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:56.324796 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:56.325293 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.325821 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.326716 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.327060 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.327178 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:56.330204 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:48969: BlockingRecv error: recv got EOF from 127.3.233.130:48969 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 16:46:56 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:56 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:56.600121 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:56.594588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:40353)
0619 16:46:56.594853 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:56.594867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:56.594983 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:56.595271 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:56.595273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:56.595447 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:56.595543 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:56.595548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.596351 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.596354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:56.596914 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:56.596923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.597382 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.597385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:56.597450 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:56.598059 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:56.598071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:56.599992 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
W20260619 16:46:56.600320 13181 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:56.594732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:40353 (local address 127.3.233.190:38471)
0619 16:46:56.594865 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:56.594868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:56.594961 (+ 93us) server_negotiation.cc:407] Connection header received
0619 16:46:56.595061 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:56.595063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:56.595108 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:56.595181 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:56.595686 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.596198 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.597026 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:56.597276 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:56.597359 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:56.600143 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:40353: BlockingRecv error: recv got EOF from 127.3.233.131:40353 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 16:46:57 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:57 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:57.133311 13184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:57.127723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:39573)
0619 16:46:57.128020 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:57.128041 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:57.128182 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:57.128472 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:57.128475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:57.128665 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:57.128773 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:57.128779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.129553 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.129556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:57.130135 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:57.130143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.130612 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.130615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:57.130690 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:57.131273 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:57.131285 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:57.133197 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
W20260619 16:46:57.133517 13185 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:57.127816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:39573 (local address 127.3.233.190:38471)
0619 16:46:57.128059 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:57.128065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:57.128130 (+ 65us) server_negotiation.cc:407] Connection header received
0619 16:46:57.128283 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:57.128285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:57.128330 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:57.128404 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:57.128890 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.129447 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.130253 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.130484 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.130566 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:57.133367 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:39573: BlockingRecv error: recv got EOF from 127.3.233.129:39573 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
Jun 19 16:46:57 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:57 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:57.336547 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:57.331043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:47271)
0619 16:46:57.331372 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:57.331390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:57.331492 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:57.331843 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:57.331845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:57.332020 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:57.332114 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:57.332118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.332889 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.332892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:57.333465 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:57.333473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.333931 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.333934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:57.333998 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:57.334521 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:57.334531 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:57.336435 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
W20260619 16:46:57.336827 13185 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:57.331466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:47271 (local address 127.3.233.190:38471)
0619 16:46:57.331634 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:57.331637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:57.331650 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:46:57.331684 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:57.331686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:57.331728 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:57.331807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:57.332229 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.332780 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.333576 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.333823 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.333905 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:57.336678 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:47271: BlockingRecv error: recv got EOF from 127.3.233.130:47271 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:46:57 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:57 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:57.606235 13187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:57.600974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:41529)
0619 16:46:57.601238 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:57.601251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:57.601344 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:57.601543 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:57.601545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:57.601721 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:57.601809 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:57.601813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.602533 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.602544 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:57.603196 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:57.603218 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.603662 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.603665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:57.603741 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:57.604241 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:57.604252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:57.606055 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
W20260619 16:46:57.606267 13185 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:57.601012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:41529 (local address 127.3.233.190:38471)
0619 16:46:57.601146 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:57.601149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:57.601317 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:46:57.601420 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:57.601422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:57.601465 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:57.601531 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:57.601900 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.602410 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.603324 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:57.603534 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:57.603612 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:57.606167 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:41529: BlockingRecv error: recv got EOF from 127.3.233.131:41529 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:46:58 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:58 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:58.139649 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:58.134060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:49657)
0619 16:46:58.134398 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:58.134416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:58.134516 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:58.134786 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:58.134788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:58.135022 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:58.135114 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:58.135120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.135921 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.135924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:58.136634 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:58.136644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.137063 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.137065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:58.137125 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:58.137637 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:58.137647 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:58.139537 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
W20260619 16:46:58.139885 13189 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:58.134298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:49657 (local address 127.3.233.190:38471)
0619 16:46:58.134540 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:58.134544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:58.134574 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:46:58.134608 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:58.134610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:58.134650 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:58.134766 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:58.135257 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.135796 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.136755 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.136969 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.137044 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:58.139745 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:49657: BlockingRecv error: recv got EOF from 127.3.233.129:49657 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
Jun 19 16:46:58 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:58 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:58.348237 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:58.337362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:43247)
0619 16:46:58.337616 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:58.337631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:58.337719 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:58.337952 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:58.337954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:58.338131 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:58.338274 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:58.338279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.339072 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.339076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:58.339664 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:58.339674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.340134 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.340137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:58.340221 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:58.346099 (+ 5878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:58.346121 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:58.348058 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
W20260619 16:46:58.348326 13189 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:58.337496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:43247 (local address 127.3.233.190:38471)
0619 16:46:58.337642 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:58.337645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:58.337697 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:46:58.337798 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:58.337800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:58.337842 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:58.337913 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:58.338393 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.338943 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.339797 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.340019 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.340127 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:58.348209 (+ 8082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:43247: BlockingRecv error: recv got EOF from 127.3.233.130:43247 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 16:46:58 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:58 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:58.613009 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:58.607264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:50951)
0619 16:46:58.607525 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:58.607539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:58.607661 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:58.607901 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:58.607903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:58.608098 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:58.608204 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:58.608209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.608953 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.608957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:58.609651 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:58.609661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.610117 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.610121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:58.610195 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:58.610764 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:58.610777 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:58.612897 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
W20260619 16:46:58.613226 13189 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:58.607320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:50951 (local address 127.3.233.190:38471)
0619 16:46:58.607442 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:58.607446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:58.607614 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:46:58.607747 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:58.607749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:58.607792 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:58.607851 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:58.608320 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.608844 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.609772 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:58.609988 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:58.610071 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:58.613079 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:50951: BlockingRecv error: recv got EOF from 127.3.233.131:50951 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 16:46:59 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:59 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:59.146492 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:59.140553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:39017)
0619 16:46:59.140903 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:59.140916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:59.141053 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:59.141409 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:59.141415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:59.141677 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:59.141775 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:59.141781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.142641 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.142644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:59.143379 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:59.143389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.143896 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.143899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:59.143967 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:59.144472 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:59.144482 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:59.146378 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":91,"threads_started":1}
W20260619 16:46:59.146656 13192 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:59.140577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:39017 (local address 127.3.233.190:38471)
0619 16:46:59.140869 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:59.140874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:59.140994 (+ 120us) server_negotiation.cc:407] Connection header received
0619 16:46:59.141198 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:59.141202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:59.141257 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:59.141361 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:59.141909 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.142513 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.143507 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.143767 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.143856 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:59.146521 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:39017: BlockingRecv error: recv got EOF from 127.3.233.129:39017 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
Jun 19 16:46:59 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:59 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:59.354899 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:59.349166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:56475)
0619 16:46:59.349544 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:59.349558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:59.349659 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:59.349900 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:59.349902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:59.350073 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:59.350168 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:59.350173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.350883 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.350885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:59.351478 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:59.351486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.351948 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.351952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:59.352036 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:59.352678 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:59.352692 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:59.354762 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":77,"threads_started":1}
W20260619 16:46:59.355110 13192 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:59.349269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:56475 (local address 127.3.233.190:38471)
0619 16:46:59.349530 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:59.349533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:59.349629 (+ 96us) server_negotiation.cc:407] Connection header received
0619 16:46:59.349753 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:59.349756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:59.349800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:59.349871 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:59.350265 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.350790 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.351576 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.351810 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.351892 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:59.354902 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:56475: BlockingRecv error: recv got EOF from 127.3.233.130:56475 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Jun 19 16:46:59 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:46:59 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:46:59.619838 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:46:59.613861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:58243)
0619 16:46:59.614253 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 16:46:59.614272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:46:59.614399 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:46:59.614682 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:46:59.614688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:46:59.615002 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:46:59.615126 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:59.615135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.615936 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.615939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:59.616514 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:46:59.616522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.616967 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.616970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:46:59.617033 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:59.617611 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:46:59.617622 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:46:59.619686 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":82,"threads_started":1}
W20260619 16:46:59.619903 13192 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:46:59.613962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:58243 (local address 127.3.233.190:38471)
0619 16:46:59.614146 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:46:59.614149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:46:59.614386 (+ 237us) server_negotiation.cc:407] Connection header received
0619 16:46:59.614485 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:46:59.614492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:46:59.614562 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:46:59.614655 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:46:59.615249 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.615818 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.616642 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:46:59.616860 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:46:59.616943 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:46:59.619809 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:58243: BlockingRecv error: recv got EOF from 127.3.233.131:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 16:47:00 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:00 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:00.153424 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:00.147532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:42711)
0619 16:47:00.147952 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:00.147973 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:00.148101 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:00.148398 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:00.148401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:00.148615 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:00.148731 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:00.148737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.149563 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.149569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:00.150139 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:00.150147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.150610 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.150614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:00.150700 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:00.151262 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:00.151273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:00.153201 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":98,"threads_started":1}
W20260619 16:47:00.153697 13197 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:00.147665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:42711 (local address 127.3.233.190:38471)
0619 16:47:00.147991 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:00.147995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:00.148041 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:47:00.148225 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:00.148228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:00.148282 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:00.148378 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:00.148865 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.149451 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.150264 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.150500 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.150607 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:00.153427 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:42711: BlockingRecv error: recv got EOF from 127.3.233.129:42711 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":95,"threads_started":1}
Jun 19 16:47:00 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:00 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:00.361534 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:00.355711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:45153)
0619 16:47:00.356092 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:00.356106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:00.356258 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:00.356533 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:00.356536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:00.356748 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:00.356837 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:00.356843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.357655 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.357662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:00.358272 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:00.358280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.358742 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.358746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:00.358829 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:00.359448 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:00.359460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:00.361361 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":101,"threads_started":1}
W20260619 16:47:00.361629 13197 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:00.355756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:45153 (local address 127.3.233.190:38471)
0619 16:47:00.355918 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:00.355922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:00.356203 (+ 281us) server_negotiation.cc:407] Connection header received
0619 16:47:00.356371 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:00.356374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:00.356426 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:00.356514 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:00.356956 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.357542 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.358396 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.358624 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.358734 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:00.361508 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:45153: BlockingRecv error: recv got EOF from 127.3.233.130:45153 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:47:00 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:00 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:00.626988 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:00.620661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:46653)
0619 16:47:00.621040 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:00.621054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:00.621149 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:00.621398 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:00.621400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:00.621594 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:00.621681 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:00.621686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.622476 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.622479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:00.623106 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:00.623115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.623699 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.623703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:00.623801 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:00.624416 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:00.624431 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:00.626802 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
W20260619 16:47:00.627209 13197 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:00.620815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:46653 (local address 127.3.233.190:38471)
0619 16:47:00.620948 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:00.620951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:00.621128 (+ 177us) server_negotiation.cc:407] Connection header received
0619 16:47:00.621232 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:00.621235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:00.621286 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:00.621361 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:00.621788 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.622381 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.623255 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:00.623575 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:00.623673 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:00.626995 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:46653: BlockingRecv error: recv got EOF from 127.3.233.131:46653 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 16:47:01 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:01 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:01.160439 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:01.154284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:41747)
0619 16:47:01.154565 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:01.154580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:01.154671 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:01.155068 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:01.155070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:01.155297 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:01.155396 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:01.155402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.156281 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.156284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:01.156915 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:01.156924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.157403 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.157408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:01.157482 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:01.158136 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:01.158158 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:01.160267 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
W20260619 16:47:01.160498 13201 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:01.154499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:41747 (local address 127.3.233.190:38471)
0619 16:47:01.154812 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:01.154816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:01.154830 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:47:01.154881 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:01.154883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:01.154948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:01.155036 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:01.155529 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.156125 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.157028 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.157258 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.157341 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:01.160384 (+ 3043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:41747: BlockingRecv error: recv got EOF from 127.3.233.129:41747 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":129,"threads_started":1}
Jun 19 16:47:01 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:01 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:01.368301 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:01.362377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:58991)
0619 16:47:01.362675 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:01.362694 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:01.362847 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:01.363157 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:01.363160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:01.363371 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:01.363465 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:01.363470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.364399 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.364404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:01.364975 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:01.364984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.365466 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.365470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:01.365563 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:01.366115 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:01.366127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:01.368171 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
W20260619 16:47:01.368505 13201 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:01.362437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:58991 (local address 127.3.233.190:38471)
0619 16:47:01.362641 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:01.362645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:01.362787 (+ 142us) server_negotiation.cc:407] Connection header received
0619 16:47:01.362971 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:01.362975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:01.363028 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:01.363131 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:01.363587 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.364284 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.365096 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.365335 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.365443 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:01.368333 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:58991: BlockingRecv error: recv got EOF from 127.3.233.130:58991 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 19 16:47:01 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:01 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:01.633641 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:01.627904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:53189)
0619 16:47:01.628175 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:01.628193 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:01.628339 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:01.628607 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:01.628609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:01.628823 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:01.628914 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:01.628920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.629697 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.629700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:01.630258 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:01.630265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.630723 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.630727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:01.630814 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:01.631411 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:01.631423 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:01.633498 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
W20260619 16:47:01.633913 13201 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:01.627958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:53189 (local address 127.3.233.190:38471)
0619 16:47:01.628114 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:01.628119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:01.628280 (+ 161us) server_negotiation.cc:407] Connection header received
0619 16:47:01.628438 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:01.628442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:01.628491 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:01.628584 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:01.629029 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.629562 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.630374 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:01.630614 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:01.630722 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:01.633650 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:53189: BlockingRecv error: recv got EOF from 127.3.233.131:53189 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 16:47:02 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:02 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:02.167158 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:02.161203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:60171)
0619 16:47:02.161478 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:02.161491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:02.161578 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:02.161829 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:02.161831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:02.162023 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:02.162124 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:02.162131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.163090 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.163093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:02.163731 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:02.163741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.164205 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.164207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:02.164302 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:02.164818 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:02.164830 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:02.166970 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
W20260619 16:47:02.167241 13205 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:02.161317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:60171 (local address 127.3.233.190:38471)
0619 16:47:02.161556 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:02.161560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:02.161576 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:47:02.161659 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:02.161661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:02.161704 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:02.161792 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:02.162359 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.162972 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.163853 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.164100 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.164186 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:02.167104 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:60171: BlockingRecv error: recv got EOF from 127.3.233.129:60171 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
Jun 19 16:47:02 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:02 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:02.374817 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:02.369130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:46789)
0619 16:47:02.369413 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:02.369432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:02.369549 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:02.369842 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:02.369844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:02.370096 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:02.370213 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:02.370220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.371053 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.371056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:02.371803 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:02.371813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.372223 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.372226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:02.372293 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:02.372807 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:02.372819 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:02.374667 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":169,"threads_started":1}
W20260619 16:47:02.374904 13205 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:02.369323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:46789 (local address 127.3.233.190:38471)
0619 16:47:02.369468 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:02.369471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:02.369499 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:47:02.369635 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:02.369637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:02.369681 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:02.369805 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:02.370321 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.370908 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.371914 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.372127 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.372204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:02.374816 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:46789: BlockingRecv error: recv got EOF from 127.3.233.130:46789 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 16:47:02 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:02 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:02.640436 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:02.634508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:42683)
0619 16:47:02.634857 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:02.634874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:02.635055 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:02.635334 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:02.635337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:02.635546 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:02.635636 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:02.635641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.636422 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.636429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:02.637001 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:02.637009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.637454 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.637457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:02.637544 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:02.638083 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:02.638097 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:02.640238 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":85,"threads_started":1}
W20260619 16:47:02.640547 13205 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:02.634646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:42683 (local address 127.3.233.190:38471)
0619 16:47:02.634858 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:02.634865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:02.634972 (+ 107us) server_negotiation.cc:407] Connection header received
0619 16:47:02.635161 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:02.635165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:02.635218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:02.635313 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:02.635748 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.636305 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.637119 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:02.637344 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:02.637453 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:02.640396 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:42683: BlockingRecv error: recv got EOF from 127.3.233.131:42683 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 16:47:03 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:03 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:03.178853 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:03.167991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:44805)
0619 16:47:03.168369 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:03.168383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:03.168479 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:03.168813 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:03.168821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:03.169007 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:03.169106 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:03.169112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.170002 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.170005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:03.170616 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:03.170625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.171105 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.171107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:03.171225 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:03.176252 (+ 5027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:03.176265 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:03.178611 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":111,"threads_started":1}
W20260619 16:47:03.178957 13209 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:03.168113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:44805 (local address 127.3.233.190:38471)
0619 16:47:03.168356 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:03.168360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:03.168456 (+ 96us) server_negotiation.cc:407] Connection header received
0619 16:47:03.168603 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:03.168606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:03.168652 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:03.168743 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:03.169344 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.169880 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.170730 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.171004 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.171086 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:03.178779 (+ 7693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:44805: BlockingRecv error: recv got EOF from 127.3.233.129:44805 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
Jun 19 16:47:03 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:03 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:03.388589 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:03.375799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:55111)
0619 16:47:03.376054 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:03.376067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:03.376154 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:03.376401 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:03.376403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:03.376575 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:03.376668 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:03.376673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.377451 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.377454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:03.378043 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:03.378051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.378610 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.378617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:03.378717 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:03.386381 (+ 7664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:03.386396 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:03.388466 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
W20260619 16:47:03.388806 13209 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:03.375914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:55111 (local address 127.3.233.190:38471)
0619 16:47:03.376095 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:03.376098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:03.376115 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:47:03.376244 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:03.376246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:03.376290 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:03.376359 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:03.376773 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.377323 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.378150 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.378473 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.378576 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:03.388659 (+ 10083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:55111: BlockingRecv error: recv got EOF from 127.3.233.130:55111 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 16:47:03 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:03 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:03.646651 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:03.641266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:59625)
0619 16:47:03.641541 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:03.641555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:03.641648 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:03.641894 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:03.641897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:03.642080 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:03.642177 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:03.642182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.642968 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.642971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:03.643610 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:03.643620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.644067 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.644070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:03.644140 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:03.644663 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:03.644675 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:03.646523 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
W20260619 16:47:03.646739 13209 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:03.641416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:59625 (local address 127.3.233.190:38471)
0619 16:47:03.641543 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:03.641548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:03.641619 (+ 71us) server_negotiation.cc:407] Connection header received
0619 16:47:03.641730 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:03.641732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:03.641784 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:03.641854 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:03.642277 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.642838 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.643724 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:03.643958 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:03.644040 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:03.646648 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:59625: BlockingRecv error: recv got EOF from 127.3.233.131:59625 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 16:47:04 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:04 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:04.185380 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:04.179698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:51273)
0619 16:47:04.179975 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:04.179989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:04.180084 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:04.180424 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:04.180427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:04.180624 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:04.180724 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:04.180729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.181549 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.181552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:04.182109 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:04.182117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.182599 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.182603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:04.182668 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:04.183318 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:04.183330 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:04.185217 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":111,"threads_started":1}
W20260619 16:47:04.185506 13213 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:04.179743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:51273 (local address 127.3.233.190:38471)
0619 16:47:04.179992 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:04.179995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:04.180058 (+ 63us) server_negotiation.cc:407] Connection header received
0619 16:47:04.180219 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:04.180222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:04.180271 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:04.180363 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:04.180867 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.181435 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.182277 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.182498 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.182588 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:04.185361 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:51273: BlockingRecv error: recv got EOF from 127.3.233.129:51273 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
Jun 19 16:47:04 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:04 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:04.395725 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:04.389620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:34901)
0619 16:47:04.389906 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:04.389920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:04.390021 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:04.390375 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:04.390379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:04.390617 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:04.390743 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:04.390753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.391714 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.391717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:04.392270 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:04.392279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.392761 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.392764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:04.392844 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:04.393453 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:04.393465 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:04.395609 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
W20260619 16:47:04.395933 13213 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:04.389745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:34901 (local address 127.3.233.190:38471)
0619 16:47:04.389912 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:04.389917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:04.390006 (+ 89us) server_negotiation.cc:407] Connection header received
0619 16:47:04.390141 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:04.390144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:04.390191 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:04.390300 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:04.390970 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.391616 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.392411 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.392620 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.392700 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:04.395792 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:34901: BlockingRecv error: recv got EOF from 127.3.233.130:34901 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 16:47:04 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:04 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:04.653273 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:04.647622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:42953)
0619 16:47:04.647908 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:04.647920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:04.648053 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:04.648344 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:04.648347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:04.648546 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:04.648633 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:04.648638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.649401 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.649406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:04.649969 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:04.649976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.650432 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.650435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:04.650524 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:04.651143 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:04.651156 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:04.653099 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
W20260619 16:47:04.653421 13213 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:04.647687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:42953 (local address 127.3.233.190:38471)
0619 16:47:04.647825 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:04.647830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:04.647992 (+ 162us) server_negotiation.cc:407] Connection header received
0619 16:47:04.648150 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:04.648154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:04.648207 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:04.648307 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:04.648771 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.649296 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.650084 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:04.650310 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:04.650417 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:04.653305 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:42953: BlockingRecv error: recv got EOF from 127.3.233.131:42953 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 16:47:05 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:05 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:05.192176 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:05.186154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:45681)
0619 16:47:05.186519 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:05.186544 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:05.186700 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:05.187018 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:05.187021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:05.187244 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:05.187337 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:05.187343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.188126 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.188131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:05.188701 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:05.188708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.189216 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.189220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:05.189305 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:05.190001 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:05.190035 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:05.192057 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":141,"threads_started":1}
W20260619 16:47:05.192411 13217 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:05.186236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:45681 (local address 127.3.233.190:38471)
0619 16:47:05.186520 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:05.186526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:05.186635 (+ 109us) server_negotiation.cc:407] Connection header received
0619 16:47:05.186811 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:05.186814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:05.186866 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:05.186996 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:05.187459 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.188016 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.188820 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.189056 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.189206 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:05.192253 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:45681: BlockingRecv error: recv got EOF from 127.3.233.129:45681 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
Jun 19 16:47:05 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:05 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:05.402486 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:05.396539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:56581)
0619 16:47:05.396867 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:05.396885 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:05.396989 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:05.397340 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:05.397342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:05.397536 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:05.397644 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:05.397649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.398434 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.398437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:05.399092 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:05.399100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.399662 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.399676 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:05.399756 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:05.400391 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:05.400405 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:05.402362 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":65,"threads_started":1}
W20260619 16:47:05.402695 13217 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:05.396757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:56581 (local address 127.3.233.190:38471)
0619 16:47:05.396909 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:05.396936 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:05.396952 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:47:05.397126 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:05.397130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:05.397193 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:05.397302 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:05.397755 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.398304 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.399219 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.399517 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.399639 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:05.402556 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:56581: BlockingRecv error: recv got EOF from 127.3.233.130:56581 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:47:05 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:05 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:05.660204 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:05.654211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:53575)
0619 16:47:05.654574 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:05.654589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:05.654691 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:05.654990 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:05.654993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:05.655174 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:05.655280 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:05.655285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.656108 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.656112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:05.656908 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:05.656919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.657455 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.657463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:05.657551 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:05.658152 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:05.658165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:05.660092 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
W20260619 16:47:05.660391 13217 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:05.654278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:53575 (local address 127.3.233.190:38471)
0619 16:47:05.654394 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:05.654398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:05.654688 (+ 290us) server_negotiation.cc:407] Connection header received
0619 16:47:05.654777 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:05.654779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:05.654825 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:05.654897 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:05.655405 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.655999 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.657035 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:05.657258 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:05.657346 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:05.660244 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:53575: BlockingRecv error: recv got EOF from 127.3.233.131:53575 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 16:47:06 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:06 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:06.198738 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:06.193023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:53441)
0619 16:47:06.193302 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:06.193316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:06.193404 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:06.193708 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:06.193711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:06.193892 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:06.194008 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:06.194014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.194883 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.194886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:06.195604 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:06.195614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.196092 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.196095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:06.196164 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:06.196746 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:06.196758 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:06.198622 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
W20260619 16:47:06.198916 13221 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:06.193075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:53441 (local address 127.3.233.190:38471)
0619 16:47:06.193349 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:06.193353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:06.193365 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:47:06.193523 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:06.193525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:06.193567 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:06.193645 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:06.194152 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.194747 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.195728 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.195960 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.196048 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:06.198768 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:53441: BlockingRecv error: recv got EOF from 127.3.233.129:53441 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
Jun 19 16:47:06 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:06 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:06.410097 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:06.403484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:50759)
0619 16:47:06.403796 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:06.403810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:06.403934 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:06.404302 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:06.404304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:06.404596 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:06.404721 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:06.404730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.405885 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.405888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:06.406526 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:06.406535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.407095 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.407098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:06.407168 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:06.407776 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:06.407790 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:06.409906 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
W20260619 16:47:06.410177 13221 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:06.403643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:50759 (local address 127.3.233.190:38471)
0619 16:47:06.403838 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:06.403842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:06.403919 (+ 77us) server_negotiation.cc:407] Connection header received
0619 16:47:06.404018 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:06.404023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:06.404095 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:06.404169 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:06.404885 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.405764 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.406668 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.406967 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.407058 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:06.410057 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:50759: BlockingRecv error: recv got EOF from 127.3.233.130:50759 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 16:47:06 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:06 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:06.666853 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:06.661049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:42929)
0619 16:47:06.661318 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:06.661335 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:06.661467 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:06.661720 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:06.661723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:06.661926 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:06.662027 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:06.662032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.662858 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.662861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:06.663576 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:06.663586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.664079 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.664082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:06.664156 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:06.664785 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:06.664797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:06.666715 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260619 16:47:06.667019 13221 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:06.661108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:42929 (local address 127.3.233.190:38471)
0619 16:47:06.661266 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:06.661271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:06.661440 (+ 169us) server_negotiation.cc:407] Connection header received
0619 16:47:06.661543 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:06.661545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:06.661605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:06.661682 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:06.662140 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.662729 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.663715 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:06.663944 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:06.664043 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:06.666868 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:42929: BlockingRecv error: recv got EOF from 127.3.233.131:42929 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:47:07 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:07 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:07.205315 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:07.199687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:59241)
0619 16:47:07.200019 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:07.200037 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:07.200152 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:07.200424 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:07.200427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:07.200614 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:07.200710 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:07.200715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.201497 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.201500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:07.202193 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:07.202202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.202654 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.202657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:07.202719 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:07.203339 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:07.203350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:07.205195 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
W20260619 16:47:07.205574 13225 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:07.199841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:59241 (local address 127.3.233.190:38471)
0619 16:47:07.200201 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:07.200205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:07.200218 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:47:07.200257 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:07.200259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:07.200300 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:07.200380 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:07.200819 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.201391 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.202312 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.202545 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.202632 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:07.205417 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:59241: BlockingRecv error: recv got EOF from 127.3.233.129:59241 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":92,"threads_started":1}
Jun 19 16:47:07 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:07 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:07.416889 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:07.410872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:40691)
0619 16:47:07.411331 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:07.411344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:07.411476 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:07.411752 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:07.411756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:07.411995 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:07.412086 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:07.412091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.412908 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.412914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:07.413488 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:07.413496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.413968 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.413972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:07.414056 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:07.414707 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:07.414718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:07.416737 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":90,"threads_started":1}
W20260619 16:47:07.417073 13225 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:07.411056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:40691 (local address 127.3.233.190:38471)
0619 16:47:07.411238 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:07.411243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:07.411423 (+ 180us) server_negotiation.cc:407] Connection header received
0619 16:47:07.411581 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:07.411585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:07.411641 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:07.411728 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:07.412208 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.412786 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.413611 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.413851 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.413965 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:07.416907 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:40691: BlockingRecv error: recv got EOF from 127.3.233.130:40691 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:47:07 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:07 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.131@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:07.674367 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:07.667941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.131:43987)
0619 16:47:07.668226 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:07.668350 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:07.668493 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:07.668789 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:07.668793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:07.669063 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:07.669166 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:07.669171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.670289 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.670293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:07.670875 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:07.670884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.671495 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.671498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:07.671564 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:07.672098 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:07.672110 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:07.674239 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
W20260619 16:47:07.674593 13225 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:07.668063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:43987 (local address 127.3.233.190:38471)
0619 16:47:07.668208 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:07.668213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:07.668486 (+ 273us) server_negotiation.cc:407] Connection header received
0619 16:47:07.668574 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:07.668577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:07.668640 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:07.668748 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:07.669373 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.670155 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.671144 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:07.671388 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:07.671473 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:07.674431 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.131:43987: BlockingRecv error: recv got EOF from 127.3.233.131:43987 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:47:08 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:08 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.129@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:08.211890 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:08.206259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.129:45415)
0619 16:47:08.206604 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:08.206617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:08.206714 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:08.207027 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:08.207029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:08.207257 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:08.207374 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.207380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.208209 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.208212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.208784 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.208792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.209238 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.209241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.209306 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.209871 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:08.209883 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:08.211764 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":161,"threads_started":1}
W20260619 16:47:08.212149 13228 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:08.206302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:45415 (local address 127.3.233.190:38471)
0619 16:47:08.206564 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:08.206569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:08.206667 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:47:08.206777 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:08.206779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:08.206843 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:08.206978 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:08.207554 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.208096 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.208903 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.209131 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.209217 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.211971 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.129:45415: BlockingRecv error: recv got EOF from 127.3.233.129:45415 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
Jun 19 16:47:08 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:47:08 dist-test-slave-cgwx krb5kdc[12170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.233.130@KRBTEST.COM for oryx/127.3.233.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:47:08.427248 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:08.417964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:38471 (local address 127.3.233.130:33003)
0619 16:47:08.418367 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:08.418382 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:08.418519 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:08.418834 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:08.418837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:08.419095 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:08.419192 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.419198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.420062 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.420068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.420713 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:08.420721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.421259 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.421264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:08.421365 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.424844 (+ 3479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:08.424856 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:08.427111 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.233.190:38471: Server oryx/127.3.233.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":94,"threads_started":1}
W20260619 16:47:08.427438 13228 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:08.418019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:33003 (local address 127.3.233.190:38471)
0619 16:47:08.418210 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:08.418215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:08.418472 (+ 257us) server_negotiation.cc:407] Connection header received
0619 16:47:08.418652 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:08.418655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:08.418707 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:08.418814 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:08.419334 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.419940 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.420892 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.421136 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.421256 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.427282 (+ 6026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.233.130:33003: BlockingRecv error: recv got EOF from 127.3.233.130:33003 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:47:08.558809 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 12753
I20260619 16:47:08.564234 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 12906
I20260619 16:47:08.569751 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13038
I20260619 16:47:08.575207 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 12683
2026-06-19T16:47:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18081 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T16:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:08Z Disabled control of system clock
I20260619 16:47:08.606144 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:46737
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:46737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:47:08.714316 13234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:08.714591 13234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:08.714627 13234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:08.718606 13234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:08.718669 13234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:08.718685 13234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:08.718701 13234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:08.718717 13234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:08.723455 13234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:46737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:08.724510 13234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:08.725828 13234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:08.734750 13239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.734750 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.734833 13240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:08.734807 13234 server_base.cc:1034] running on GCE node
I20260619 16:47:08.735576 13234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:08.736773 13234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:08.737968 13234 hybrid_clock.cc:648] HybridClock initialized: now 1781887628737949 us; error 29 us; skew 500 ppm
I20260619 16:47:08.740300 13234 webserver.cc:460] Webserver started at http://127.3.233.190:39735/ using document root <none> and password file <none>
I20260619 16:47:08.740963 13234 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:08.741034 13234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:08.741309 13234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:08.743683 13234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c"
format_stamp: "Formatted at 2026-06-19 16:47:08 on dist-test-slave-cgwx"
server_key: "abd17b6e3af5d808d60d71c080a6f7c9"
server_key_iv: "511acca439e9a892af36e9f12cc129df"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.744306 13234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c"
format_stamp: "Formatted at 2026-06-19 16:47:08 on dist-test-slave-cgwx"
server_key: "abd17b6e3af5d808d60d71c080a6f7c9"
server_key_iv: "511acca439e9a892af36e9f12cc129df"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.748584 13234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:47:08.751127 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.752514 13234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:08.752656 13234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c"
format_stamp: "Formatted at 2026-06-19 16:47:08 on dist-test-slave-cgwx"
server_key: "abd17b6e3af5d808d60d71c080a6f7c9"
server_key_iv: "511acca439e9a892af36e9f12cc129df"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.752775 13234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:08.773552 13234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:08.774291 13234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:08.774508 13234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:08.781466 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:46737 every 8 connection(s)
I20260619 16:47:08.781467 13234 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:46737
I20260619 16:47:08.782548 13234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:08.785951 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:08.791653 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c: Bootstrap starting.
I20260619 16:47:08.792173 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13234
I20260619 16:47:08.792369 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:47:08.792659 4006 external_mini_cluster.cc:1428] Setting key 81fb514410dff222fc275beaaa8cdde3
I20260619 16:47:08.794564 13301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:08.795411 13301 log.cc:824] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:08.797334 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c: No bootstrap required, opened a new log
I20260619 16:47:08.800932 13301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46737 } }
I20260619 16:47:08.801144 13301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:08.801213 13301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8c9ba1aaae4689bfc6fe65368bde8c, State: Initialized, Role: FOLLOWER
I20260619 16:47:08.801745 13301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46737 } }
I20260619 16:47:08.801877 13301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:08.801983 13301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:08.802107 13301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:08.802621 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:08.794011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.3.233.190:46737)
0619 16:47:08.794864 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:08.794880 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:08.794973 (+ 93us) server_negotiation.cc:407] Connection header received
0619 16:47:08.795688 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:08.795706 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:08.796082 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:08.796425 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:08.796869 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.797926 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.798793 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:08.799434 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:08.799601 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:08.800583 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:08.800589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:08.800624 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:08.801029 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:08.801085 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:08.801106 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:08.801296 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:08.801412 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:47:08.801681 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":136,"threads_started":1}
I20260619 16:47:08.803184 13301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46737 } }
I20260619 16:47:08.803563 13301 leader_election.cc:302] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8c9ba1aaae4689bfc6fe65368bde8c; no voters:
I20260619 16:47:08.803884 13301 leader_election.cc:288] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:08.803963 13306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:08.804158 13306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [term 1 LEADER]: Becoming Leader. State: Replica: 1b8c9ba1aaae4689bfc6fe65368bde8c, State: Running, Role: LEADER
I20260619 16:47:08.804493 13306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46737 } }
I20260619 16:47:08.805145 13301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:08.806262 13307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46737 } } }
I20260619 16:47:08.806401 13307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:08.806717 13308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8c9ba1aaae4689bfc6fe65368bde8c. Latest consensus state: current_term: 1 leader_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8c9ba1aaae4689bfc6fe65368bde8c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 46737 } } }
I20260619 16:47:08.806787 13315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:08.809636 13315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:08.811645 13308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:08.815237 13315 catalog_manager.cc:1347] Generated new cluster ID: 1a75fc80f7ee4084a0b0f4389f2fbebc
I20260619 16:47:08.815304 13315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:08.823132 13315 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:08.824307 13315 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:08.831032 13315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b8c9ba1aaae4689bfc6fe65368bde8c: Generated new TSK 0
I20260619 16:47:08.831759 13315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:47:08.839972 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46737
--builtin_ntp_servers=127.3.233.148:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 16:47:08.951229 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:08.951607 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:08.951686 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:08.956131 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:08.956192 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:08.956249 13325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:47:08.962278 13325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:08.963529 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:08.965011 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:08.976789 13331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.976802 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:08.976785 13330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:08.978328 13325 server_base.cc:1034] running on GCE node
I20260619 16:47:08.978775 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:08.979476 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:08.980676 13325 hybrid_clock.cc:648] HybridClock initialized: now 1781887628980665 us; error 28 us; skew 500 ppm
I20260619 16:47:08.982682 13325 webserver.cc:460] Webserver started at http://127.3.233.129:44353/ using document root <none> and password file <none>
I20260619 16:47:08.983350 13325 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:08.983446 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:08.983680 13325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:08.985733 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "6f1462d1976c46e1b0c81d7c3f8459dc"
format_stamp: "Formatted at 2026-06-19 16:47:08 on dist-test-slave-cgwx"
server_key: "fbee9dc85bfdc9974d01166bfb6d409e"
server_key_iv: "bb47b2264d00dead1a7bb75a6161ca99"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.986321 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "6f1462d1976c46e1b0c81d7c3f8459dc"
format_stamp: "Formatted at 2026-06-19 16:47:08 on dist-test-slave-cgwx"
server_key: "fbee9dc85bfdc9974d01166bfb6d409e"
server_key_iv: "bb47b2264d00dead1a7bb75a6161ca99"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.989965 13325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:47:08.992650 13339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:08.993917 13325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:47:08.994048 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "6f1462d1976c46e1b0c81d7c3f8459dc"
format_stamp: "Formatted at 2026-06-19 16:47:08 on dist-test-slave-cgwx"
server_key: "fbee9dc85bfdc9974d01166bfb6d409e"
server_key_iv: "bb47b2264d00dead1a7bb75a6161ca99"
server_key_version: "encryptionkey@0"
I20260619 16:47:08.994164 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:09.015991 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:09.016767 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:09.016980 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:09.017617 13325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:09.018630 13325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:09.018703 13325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.018774 13325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:09.018827 13325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.028337 13325 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:38093
I20260619 16:47:09.028378 13452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:38093 every 8 connection(s)
I20260619 16:47:09.029295 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:09.036612 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13325
I20260619 16:47:09.036763 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:09.037138 4006 external_mini_cluster.cc:1428] Setting key d1c4b7e271d7e3bd672b3c41d1476ab4
I20260619 16:47:09.039418 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.031169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:58641 (local address 127.3.233.190:46737)
0619 16:47:09.031378 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.031382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.032550 (+ 1168us) server_negotiation.cc:407] Connection header received
0619 16:47:09.034029 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.034032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.034081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.034169 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.035231 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.035787 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.037044 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.037397 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.037533 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.038855 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.038859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.038862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:09.038875 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.038896 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:09.038906 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.039029 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.039167 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.039230 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:47:09.039495 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46737
--builtin_ntp_servers=127.3.233.148:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:47:09.040303 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.031661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:46737 (local address 127.3.233.129:58641)
0619 16:47:09.032396 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.032439 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.033803 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.034341 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.034348 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.034487 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:09.035078 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.035094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.035908 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.035911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.036907 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.036916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.037575 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.037580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.037786 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.038510 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:09.038512 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:09.038580 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:09.038670 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:09.039031 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:09.039038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:09.039042 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.039145 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.039411 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":109,"threads_started":1}
I20260619 16:47:09.041630 13453 heartbeater.cc:347] Connected to a master server at 127.3.233.190:46737
I20260619 16:47:09.041867 13453 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:09.042397 13453 heartbeater.cc:511] Master 127.3.233.190:46737 requested a full tablet report, sending...
I20260619 16:47:09.043933 13265 ts_manager.cc:194] Registered new tserver with Master: 6f1462d1976c46e1b0c81d7c3f8459dc (127.3.233.129:38093)
I20260619 16:47:09.045120 13265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.129:58641
W20260619 16:47:09.149854 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:09.150161 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:09.150231 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:09.154485 13456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:09.154587 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:09.154675 13456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:47:09.159788 13456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:09.160887 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:09.162298 13456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:09.171717 13464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.171691 13461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.171691 13462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:09.172006 13456 server_base.cc:1034] running on GCE node
I20260619 16:47:09.172431 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:09.173082 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:09.174278 13456 hybrid_clock.cc:648] HybridClock initialized: now 1781887629174245 us; error 47 us; skew 500 ppm
I20260619 16:47:09.176219 13456 webserver.cc:460] Webserver started at http://127.3.233.130:45705/ using document root <none> and password file <none>
I20260619 16:47:09.176790 13456 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:09.176890 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:09.177124 13456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:09.179240 13456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "695d5125ed814fd1990908c599f0cfbd"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "c78c3512057d3e028bd6d41ac2214b56"
server_key_iv: "b9c3c58592ca4b06cc344fa8a03a53e7"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.179785 13456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "695d5125ed814fd1990908c599f0cfbd"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "c78c3512057d3e028bd6d41ac2214b56"
server_key_iv: "b9c3c58592ca4b06cc344fa8a03a53e7"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.183421 13456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:47:09.185930 13470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.187381 13456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:47:09.187536 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "695d5125ed814fd1990908c599f0cfbd"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "c78c3512057d3e028bd6d41ac2214b56"
server_key_iv: "b9c3c58592ca4b06cc344fa8a03a53e7"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.187664 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:09.202991 13456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:09.203765 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:09.203980 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:09.204598 13456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:09.205703 13456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:09.205782 13456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.205848 13456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:09.205901 13456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.215267 13456 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:44425
I20260619 16:47:09.215287 13583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:44425 every 8 connection(s)
I20260619 16:47:09.216259 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:09.226287 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.218919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:41527 (local address 127.3.233.190:46737)
0619 16:47:09.219100 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.219104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.220258 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 16:47:09.221636 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.221639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.221685 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.221766 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.222693 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.223266 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.224397 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.224614 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.224693 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.225751 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.225754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.225756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:09.225768 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.225785 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:09.225793 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.225850 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.226055 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.226122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:47:09.226287 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13456
I20260619 16:47:09.226398 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:09.226727 4006 external_mini_cluster.cc:1428] Setting key eda61f382f571428a1fcfe30e80b617c
I20260619 16:47:09.227274 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.219374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:46737 (local address 127.3.233.130:41527)
0619 16:47:09.220107 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.220150 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.221418 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.221962 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.221969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.222055 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:09.222549 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.222562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.223378 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.223381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.224291 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.224300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.224743 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.224746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.224914 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.225509 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:09.225511 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:09.225549 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:09.225614 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:09.225864 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:09.225896 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:09.225901 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.225998 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.226222 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260619 16:47:09.228523 13584 heartbeater.cc:347] Connected to a master server at 127.3.233.190:46737
I20260619 16:47:09.228771 13584 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:09.228837 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46737
--builtin_ntp_servers=127.3.233.148:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:47:09.229349 13584 heartbeater.cc:511] Master 127.3.233.190:46737 requested a full tablet report, sending...
I20260619 16:47:09.230499 13263 ts_manager.cc:194] Registered new tserver with Master: 695d5125ed814fd1990908c599f0cfbd (127.3.233.130:44425)
I20260619 16:47:09.231192 13263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.130:41527
W20260619 16:47:09.343149 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:09.343484 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:09.343585 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:09.347848 13587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:09.347934 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:09.348024 13587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:47:09.352972 13587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:09.354147 13587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:09.355667 13587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:09.365516 13595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.365481 13593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:09.365537 13587 server_base.cc:1034] running on GCE node
W20260619 16:47:09.365481 13592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:09.366142 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:09.366784 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:09.367966 13587 hybrid_clock.cc:648] HybridClock initialized: now 1781887629367941 us; error 33 us; skew 500 ppm
I20260619 16:47:09.369967 13587 webserver.cc:460] Webserver started at http://127.3.233.131:41699/ using document root <none> and password file <none>
I20260619 16:47:09.370503 13587 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:09.370561 13587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:09.370720 13587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:09.372843 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "3502ec15f2cb429da1f9499a9caad062"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "6a76f8f43c6c2f2efec0d12c1b7f1fe1"
server_key_iv: "6472a7dbc2f6d92ae2c25f1209f3e9f6"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.373420 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "3502ec15f2cb429da1f9499a9caad062"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "6a76f8f43c6c2f2efec0d12c1b7f1fe1"
server_key_iv: "6472a7dbc2f6d92ae2c25f1209f3e9f6"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.377177 13587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:47:09.379827 13601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.381086 13587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:47:09.381228 13587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "3502ec15f2cb429da1f9499a9caad062"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "6a76f8f43c6c2f2efec0d12c1b7f1fe1"
server_key_iv: "6472a7dbc2f6d92ae2c25f1209f3e9f6"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.381349 13587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:09.401445 13587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:09.402180 13587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:09.402390 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:09.403053 13587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:09.404006 13587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:09.404074 13587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.404145 13587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:09.404179 13587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.413841 13587 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:35521
I20260619 16:47:09.413872 13714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:35521 every 8 connection(s)
I20260619 16:47:09.414790 13587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:09.415874 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13587
I20260619 16:47:09.415982 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:09.416265 4006 external_mini_cluster.cc:1428] Setting key 405cd2de16460504d4eafb06315535cb
I20260619 16:47:09.425241 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.417143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:34941 (local address 127.3.233.190:46737)
0619 16:47:09.417285 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.417289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.418682 (+ 1393us) server_negotiation.cc:407] Connection header received
0619 16:47:09.420165 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.420168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.420213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.420286 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.421176 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.421709 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.422990 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.423233 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.423326 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.424817 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.424820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.424822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:09.424835 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.424876 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:09.424885 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.424948 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.425070 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.425122 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:47:09.426167 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.417709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:46737 (local address 127.3.233.131:34941)
0619 16:47:09.418516 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.418594 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.419949 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.420432 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.420439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.420524 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:09.421024 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.421036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.421867 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.421870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.422798 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:09.422807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:09.423424 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:09.423428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:09.423616 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:09.424470 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:09.424474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:09.424530 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:09.424634 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:09.424963 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:09.424969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:09.424973 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.425071 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.425298 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":161,"threads_started":1}
I20260619 16:47:09.427371 13715 heartbeater.cc:347] Connected to a master server at 127.3.233.190:46737
I20260619 16:47:09.427579 13715 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:09.428053 13715 heartbeater.cc:511] Master 127.3.233.190:46737 requested a full tablet report, sending...
I20260619 16:47:09.429064 13263 ts_manager.cc:194] Registered new tserver with Master: 3502ec15f2cb429da1f9499a9caad062 (127.3.233.131:35521)
I20260619 16:47:09.429715 13263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.131:34941
I20260619 16:47:09.430658 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:47:09.433399 13304 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:09.432661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.3.233.190:46737)
0619 16:47:09.432783 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.432786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.432943 (+ 157us) server_negotiation.cc:407] Connection header received
0619 16:47:09.433038 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.433040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.433076 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.433129 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.433300 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38640: BlockingRecv error: recv got EOF from 127.0.0.1:38640 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:47:09.433916 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13325
I20260619 16:47:09.439630 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13456
I20260619 16:47:09.444921 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13587
I20260619 16:47:09.450243 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13234
2026-06-19T16:47:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (873 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T16:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:09Z Disabled control of system clock
I20260619 16:47:09.479758 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37823
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:37823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:47:09.591950 13726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:09.592235 13726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:09.592271 13726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:09.596370 13726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:09.596436 13726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:09.596451 13726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:09.596469 13726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:09.596484 13726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:09.601447 13726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:37823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:09.602571 13726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:09.603981 13726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:09.613164 13732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.613143 13731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.613174 13734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:09.613672 13726 server_base.cc:1034] running on GCE node
I20260619 16:47:09.614403 13726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:09.615814 13726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:09.617034 13726 hybrid_clock.cc:648] HybridClock initialized: now 1781887629617011 us; error 48 us; skew 500 ppm
I20260619 16:47:09.619302 13726 webserver.cc:460] Webserver started at http://127.3.233.190:34459/ using document root <none> and password file <none>
I20260619 16:47:09.619933 13726 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:09.620031 13726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:09.620270 13726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:09.622408 13726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "5a857ad9716e6ce58f039f91bbd22579"
server_key_iv: "85112fa8f0257b898598bf93b4ef71e6"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.623059 13726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "5a857ad9716e6ce58f039f91bbd22579"
server_key_iv: "85112fa8f0257b898598bf93b4ef71e6"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.627041 13726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:47:09.629596 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.630877 13726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:47:09.631047 13726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "5a857ad9716e6ce58f039f91bbd22579"
server_key_iv: "85112fa8f0257b898598bf93b4ef71e6"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.631173 13726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:09.636042 13726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:09.636725 13726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:09.636921 13726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:09.644261 13726 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:37823
I20260619 16:47:09.644285 13792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:37823 every 8 connection(s)
I20260619 16:47:09.645244 13726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:09.645737 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13726
I20260619 16:47:09.645877 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:47:09.646194 4006 external_mini_cluster.cc:1428] Setting key 70af50f35b4446cfa529b5bb91f80f53
I20260619 16:47:09.648550 13793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:09.652719 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.647750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.3.233.190:37823)
0619 16:47:09.648876 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.648893 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.648932 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:47:09.649603 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.649620 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.649978 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.650267 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.651214 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.651223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.651241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:09.651257 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.651337 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:09.651364 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.651546 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.651712 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.652005 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":588,"threads_started":1}
I20260619 16:47:09.654964 13793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8: Bootstrap starting.
I20260619 16:47:09.657163 13793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:09.657938 13793 log.cc:824] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:09.660332 13793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8: No bootstrap required, opened a new log
I20260619 16:47:09.664047 13793 raft_consensus.cc:348] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37823 } }
I20260619 16:47:09.664317 13793 raft_consensus.cc:374] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:09.664422 13793 raft_consensus.cc:729] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77ad4dc58de45029e3a0f3bd2fef5b8, State: Initialized, Role: FOLLOWER
I20260619 16:47:09.664918 13793 consensus_queue.cc:260] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37823 } }
I20260619 16:47:09.665071 13793 raft_consensus.cc:388] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:09.665156 13793 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:09.665259 13793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:09.666217 13793 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37823 } }
I20260619 16:47:09.666638 13793 leader_election.cc:302] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b77ad4dc58de45029e3a0f3bd2fef5b8; no voters:
I20260619 16:47:09.667057 13793 leader_election.cc:288] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:09.667133 13798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:09.667341 13798 raft_consensus.cc:686] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [term 1 LEADER]: Becoming Leader. State: Replica: b77ad4dc58de45029e3a0f3bd2fef5b8, State: Running, Role: LEADER
I20260619 16:47:09.667635 13798 consensus_queue.cc:237] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37823 } }
I20260619 16:47:09.668426 13793 sys_catalog.cc:564] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:09.669624 13799 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77ad4dc58de45029e3a0f3bd2fef5b8. Latest consensus state: current_term: 1 leader_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37823 } } }
I20260619 16:47:09.669749 13799 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:09.669986 13800 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77ad4dc58de45029e3a0f3bd2fef5b8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37823 } } }
I20260619 16:47:09.670075 13800 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:09.670200 13807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:09.673599 13807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:09.679629 13807 catalog_manager.cc:1347] Generated new cluster ID: 6ec604a234b4422daa8fccea3d6dda30
I20260619 16:47:09.679709 13807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:09.700608 13807 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:09.701009 13807 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:09.706494 13807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b77ad4dc58de45029e3a0f3bd2fef5b8: Generated new TSK 0
I20260619 16:47:09.707396 13807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:47:09.714732 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37823
--builtin_ntp_servers=127.3.233.148:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:47:09.833318 13817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:09.833663 13817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:09.833736 13817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:09.838204 13817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:09.838304 13817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:09.838406 13817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:47:09.843406 13817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:09.844535 13817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:09.845963 13817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:09.855769 13823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:09.855777 13822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:09.856055 13817 server_base.cc:1034] running on GCE node
W20260619 16:47:09.855769 13825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:09.856698 13817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:09.857402 13817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:09.858603 13817 hybrid_clock.cc:648] HybridClock initialized: now 1781887629858602 us; error 40 us; skew 500 ppm
I20260619 16:47:09.860790 13817 webserver.cc:460] Webserver started at http://127.3.233.129:43607/ using document root <none> and password file <none>
I20260619 16:47:09.861428 13817 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:09.861524 13817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:09.861765 13817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:09.863986 13817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "623227e1fa1047809c45304de22c7f73"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "1456c5b019cd14a0093010014810f08b"
server_key_iv: "b344955a57f91560dae4892e6deee916"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.864650 13817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "623227e1fa1047809c45304de22c7f73"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "1456c5b019cd14a0093010014810f08b"
server_key_iv: "b344955a57f91560dae4892e6deee916"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.868652 13817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:47:09.871709 13831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.873070 13817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:47:09.873229 13817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "623227e1fa1047809c45304de22c7f73"
format_stamp: "Formatted at 2026-06-19 16:47:09 on dist-test-slave-cgwx"
server_key: "1456c5b019cd14a0093010014810f08b"
server_key_iv: "b344955a57f91560dae4892e6deee916"
server_key_version: "encryptionkey@0"
I20260619 16:47:09.873364 13817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:09.878192 13817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:09.878824 13817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:09.879050 13817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:09.879612 13817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:09.880535 13817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:09.880609 13817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.880678 13817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:09.880722 13817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:09.890136 13817 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:43815
I20260619 16:47:09.890213 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:43815 every 8 connection(s)
I20260619 16:47:09.891189 13817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:09.891528 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13817
I20260619 16:47:09.891646 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:09.892066 4006 external_mini_cluster.cc:1428] Setting key 3e7cef9a33e73e8a231a3a2b623adaa1
I20260619 16:47:09.894439 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37823
--builtin_ntp_servers=127.3.233.148:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:47:09.898005 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.893169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:50517 (local address 127.3.233.190:37823)
0619 16:47:09.893311 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:09.893316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:09.894614 (+ 1298us) server_negotiation.cc:407] Connection header received
0619 16:47:09.896068 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:09.896074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:09.896120 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:09.896221 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:09.897379 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:09.897386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:09.897392 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:09.897410 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:09.897438 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:09.897451 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:09.897558 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:09.897753 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:47:09.897808 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:47:09.899042 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:09.893695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:37823 (local address 127.3.233.129:50517)
0619 16:47:09.894446 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:09.894488 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:09.895851 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:09.896378 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:09.896385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:09.896476 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:09.897107 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:09.897109 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:09.897152 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:09.897227 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:09.897584 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:09.897596 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:09.897602 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:47:09.897776 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:47:09.898030 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260619 16:47:09.900350 13945 heartbeater.cc:347] Connected to a master server at 127.3.233.190:37823
I20260619 16:47:09.900593 13945 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:09.901036 13945 heartbeater.cc:511] Master 127.3.233.190:37823 requested a full tablet report, sending...
I20260619 16:47:09.902614 13757 ts_manager.cc:194] Registered new tserver with Master: 623227e1fa1047809c45304de22c7f73 (127.3.233.129:43815)
W20260619 16:47:10.009397 13948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:10.009675 13948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:10.009717 13948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:10.014039 13948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:10.014115 13948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:10.014175 13948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:47:10.019274 13948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:10.020452 13948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:10.021847 13948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:10.031467 13956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:10.031562 13954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:10.031527 13953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:10.031731 13948 server_base.cc:1034] running on GCE node
I20260619 16:47:10.032276 13948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:10.033020 13948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:10.034227 13948 hybrid_clock.cc:648] HybridClock initialized: now 1781887630034212 us; error 40 us; skew 500 ppm
I20260619 16:47:10.036198 13948 webserver.cc:460] Webserver started at http://127.3.233.130:46585/ using document root <none> and password file <none>
I20260619 16:47:10.036803 13948 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:10.036897 13948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:10.037132 13948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:10.039261 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "a44affe1ed144a2cb83e5aaf0ca61ee8"
format_stamp: "Formatted at 2026-06-19 16:47:10 on dist-test-slave-cgwx"
server_key: "4af60d12090addf4fffe4b6223a42e9b"
server_key_iv: "e3b1e33674c9ccc736b980785195a866"
server_key_version: "encryptionkey@0"
I20260619 16:47:10.039824 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "a44affe1ed144a2cb83e5aaf0ca61ee8"
format_stamp: "Formatted at 2026-06-19 16:47:10 on dist-test-slave-cgwx"
server_key: "4af60d12090addf4fffe4b6223a42e9b"
server_key_iv: "e3b1e33674c9ccc736b980785195a866"
server_key_version: "encryptionkey@0"
I20260619 16:47:10.043661 13948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:47:10.046336 13962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:10.047804 13948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:47:10.047958 13948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "a44affe1ed144a2cb83e5aaf0ca61ee8"
format_stamp: "Formatted at 2026-06-19 16:47:10 on dist-test-slave-cgwx"
server_key: "4af60d12090addf4fffe4b6223a42e9b"
server_key_iv: "e3b1e33674c9ccc736b980785195a866"
server_key_version: "encryptionkey@0"
I20260619 16:47:10.048082 13948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:10.053102 13948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:10.053716 13948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:10.053898 13948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:10.054435 13948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:10.055406 13948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:10.055473 13948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:10.055544 13948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:10.055596 13948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:10.064670 13948 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:33533
I20260619 16:47:10.064693 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:33533 every 8 connection(s)
I20260619 16:47:10.065704 13948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:10.071313 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 13948
I20260619 16:47:10.071450 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:10.071762 4006 external_mini_cluster.cc:1428] Setting key 60dc27382320f7ded5d46148098e04b1
I20260619 16:47:10.072640 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:10.067867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:35885 (local address 127.3.233.190:37823)
0619 16:47:10.068016 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:10.068020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:10.069137 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 16:47:10.070730 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:10.070734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:10.070788 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:10.070898 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:10.072157 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:10.072161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:10.072165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:10.072178 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:10.072199 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:10.072210 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:10.072316 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:10.072463 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 16:47:10.072508 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:47:10.073582 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:10.068254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:37823 (local address 127.3.233.130:35885)
0619 16:47:10.068989 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:10.069033 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:10.070526 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:10.071085 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:10.071096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:10.071187 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:10.071887 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:10.071891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:10.071945 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:10.072014 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:10.072307 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:10.072315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:10.072321 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:47:10.072506 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:47:10.072826 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":102,"threads_started":1}
I20260619 16:47:10.073833 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37823
--builtin_ntp_servers=127.3.233.148:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:47:10.074753 14076 heartbeater.cc:347] Connected to a master server at 127.3.233.190:37823
I20260619 16:47:10.075017 14076 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:10.075420 14076 heartbeater.cc:511] Master 127.3.233.190:37823 requested a full tablet report, sending...
I20260619 16:47:10.076594 13757 ts_manager.cc:194] Registered new tserver with Master: a44affe1ed144a2cb83e5aaf0ca61ee8 (127.3.233.130:33533)
W20260619 16:47:10.186566 14079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:10.186885 14079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:10.186997 14079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:10.191617 14079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:10.191691 14079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:10.191788 14079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:47:10.196918 14079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:10.198060 14079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:10.199577 14079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:10.209456 14087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:10.209475 14085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:10.209460 14084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:10.209908 14079 server_base.cc:1034] running on GCE node
I20260619 16:47:10.210489 14079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:10.211238 14079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:10.212453 14079 hybrid_clock.cc:648] HybridClock initialized: now 1781887630212438 us; error 31 us; skew 500 ppm
I20260619 16:47:10.214470 14079 webserver.cc:460] Webserver started at http://127.3.233.131:40385/ using document root <none> and password file <none>
I20260619 16:47:10.215148 14079 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:10.215215 14079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:10.215431 14079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:10.217473 14079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "58c057eb17bc486da1e0101de8462234"
format_stamp: "Formatted at 2026-06-19 16:47:10 on dist-test-slave-cgwx"
server_key: "f0b32156d5099489c80671e3d55025f1"
server_key_iv: "8927e9da8d9ee0d8f55488a6f984ba84"
server_key_version: "encryptionkey@0"
I20260619 16:47:10.218024 14079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "58c057eb17bc486da1e0101de8462234"
format_stamp: "Formatted at 2026-06-19 16:47:10 on dist-test-slave-cgwx"
server_key: "f0b32156d5099489c80671e3d55025f1"
server_key_iv: "8927e9da8d9ee0d8f55488a6f984ba84"
server_key_version: "encryptionkey@0"
I20260619 16:47:10.221793 14079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:47:10.224440 14093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:10.225782 14079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:47:10.225880 14079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "58c057eb17bc486da1e0101de8462234"
format_stamp: "Formatted at 2026-06-19 16:47:10 on dist-test-slave-cgwx"
server_key: "f0b32156d5099489c80671e3d55025f1"
server_key_iv: "8927e9da8d9ee0d8f55488a6f984ba84"
server_key_version: "encryptionkey@0"
I20260619 16:47:10.226022 14079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:10.230652 14079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:10.231297 14079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:10.231477 14079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:10.232004 14079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:10.232908 14079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:10.232951 14079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:10.233014 14079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:10.233052 14079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:10.241788 14079 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:41373
I20260619 16:47:10.241850 14206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:41373 every 8 connection(s)
I20260619 16:47:10.242733 14079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:10.248860 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:10.244577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:43097 (local address 127.3.233.190:37823)
0619 16:47:10.244717 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:10.244721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:10.245777 (+ 1056us) server_negotiation.cc:407] Connection header received
0619 16:47:10.247144 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:10.247147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:10.247189 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:10.247265 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:10.248383 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:10.248387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:10.248390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:47:10.248402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:10.248423 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:47:10.248432 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:10.248538 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:10.248674 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:47:10.248716 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:47:10.249648 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:10.244934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:37823 (local address 127.3.233.131:43097)
0619 16:47:10.245629 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:10.245670 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:10.246868 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:10.247427 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:10.247434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:10.247521 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:10.248127 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:47:10.248128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:10.248167 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:47:10.248231 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:10.248520 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:10.248527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:10.248533 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:47:10.248714 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:47:10.248939 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260619 16:47:10.250738 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 14079
I20260619 16:47:10.250823 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:10.250945 14207 heartbeater.cc:347] Connected to a master server at 127.3.233.190:37823
I20260619 16:47:10.251180 14207 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:10.251183 4006 external_mini_cluster.cc:1428] Setting key da990b7cff23bea3e22c5bc9ff7a0fdb
I20260619 16:47:10.251729 14207 heartbeater.cc:511] Master 127.3.233.190:37823 requested a full tablet report, sending...
I20260619 16:47:10.252722 13756 ts_manager.cc:194] Registered new tserver with Master: 58c057eb17bc486da1e0101de8462234 (127.3.233.131:41373)
I20260619 16:47:10.253774 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:47:10.256723 13796 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:47:10.255633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.3.233.190:37823)
0619 16:47:10.255747 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:10.255752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:10.256103 (+ 351us) server_negotiation.cc:407] Connection header received
0619 16:47:10.256249 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:10.256251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:10.256282 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:10.256355 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:10.256636 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53260: BlockingRecv error: recv got EOF from 127.0.0.1:53260 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:47:10.256937 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13817
I20260619 16:47:10.262840 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13948
I20260619 16:47:10.267874 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 14079
I20260619 16:47:10.273068 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 13726
2026-06-19T16:47:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (822 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:47:10 dist-test-slave-cgwx krb5kdc[14217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:47:10 dist-test-slave-cgwx krb5kdc[14217](info): set up 2 sockets
Jun 19 16:47:10 dist-test-slave-cgwx krb5kdc[14217](info): commencing operation
krb5kdc: starting...
W20260619 16:47:11.328099 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:47:11 dist-test-slave-cgwx krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:47:11.449890 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35395
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:35395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:11.560853 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:11.561136 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:11.561175 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:11.565258 14230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:11.565320 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:11.565340 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:11.565357 14230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:11.565372 14230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:11.570655 14230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:35395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:11.571743 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:11.572757 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:11.581156 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:11.581310 14230 server_base.cc:1034] running on GCE node
W20260619 16:47:11.581202 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:11.581190 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:11.581991 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:11.583244 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:11.584460 14230 hybrid_clock.cc:648] HybridClock initialized: now 1781887631584443 us; error 28 us; skew 500 ppm
Jun 19 16:47:11 dist-test-slave-cgwx krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:11.587383 14230 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:47:11.588217 14230 webserver.cc:460] Webserver started at http://127.3.233.190:36663/ using document root <none> and password file <none>
I20260619 16:47:11.588727 14230 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:11.588783 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:11.588943 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:11.590899 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6"
format_stamp: "Formatted at 2026-06-19 16:47:11 on dist-test-slave-cgwx"
server_key: "44ce187f892d47c4e70f84e45c093c88"
server_key_iv: "7efcb257a7c0b9c4634b609159f794ba"
server_key_version: "encryptionkey@0"
I20260619 16:47:11.591534 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6"
format_stamp: "Formatted at 2026-06-19 16:47:11 on dist-test-slave-cgwx"
server_key: "44ce187f892d47c4e70f84e45c093c88"
server_key_iv: "7efcb257a7c0b9c4634b609159f794ba"
server_key_version: "encryptionkey@0"
I20260619 16:47:11.595363 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:47:11.597872 14245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:11.599210 14230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:47:11.599359 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6"
format_stamp: "Formatted at 2026-06-19 16:47:11 on dist-test-slave-cgwx"
server_key: "44ce187f892d47c4e70f84e45c093c88"
server_key_iv: "7efcb257a7c0b9c4634b609159f794ba"
server_key_version: "encryptionkey@0"
I20260619 16:47:11.599481 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:11.616304 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:11.619845 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:11.620064 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:11.627164 14230 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:35395
I20260619 16:47:11.627444 14297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:35395 every 8 connection(s)
I20260619 16:47:11.628199 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:11.630992 14298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:11.636353 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 14230
I20260619 16:47:11.636420 14298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6: Bootstrap starting.
I20260619 16:47:11.636509 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:47:11.636793 4006 external_mini_cluster.cc:1428] Setting key 6ee43255a3076deecd25aece762316a2
I20260619 16:47:11.639106 14298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:11.639972 14298 log.cc:824] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:11.642246 14298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6: No bootstrap required, opened a new log
I20260619 16:47:11.646224 14298 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35395 } }
I20260619 16:47:11.646448 14298 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:11.646524 14298 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb799cdc52a477c8df6c1a85ddf1ab6, State: Initialized, Role: FOLLOWER
Jun 19 16:47:11 dist-test-slave-cgwx krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:11.647089 14298 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35395 } }
I20260619 16:47:11.647276 14298 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:11.647342 14298 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:11.647449 14298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:11.648384 14298 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35395 } }
I20260619 16:47:11.648728 14298 leader_election.cc:302] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceb799cdc52a477c8df6c1a85ddf1ab6; no voters:
I20260619 16:47:11.649118 14298 leader_election.cc:288] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:11.649231 14303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:11.649534 14303 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [term 1 LEADER]: Becoming Leader. State: Replica: ceb799cdc52a477c8df6c1a85ddf1ab6, State: Running, Role: LEADER
I20260619 16:47:11.649883 14303 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35395 } }
I20260619 16:47:11.650532 14298 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:11.651491 14305 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb799cdc52a477c8df6c1a85ddf1ab6. Latest consensus state: current_term: 1 leader_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35395 } } }
I20260619 16:47:11.651635 14305 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:11.651954 14304 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb799cdc52a477c8df6c1a85ddf1ab6" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35395 } } }
I20260619 16:47:11.652047 14304 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:11.652343 14312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:11.653807 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:11.638219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.3.233.190:35395)
0619 16:47:11.638657 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:11.638676 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:11.638717 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:47:11.639641 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:11.639663 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:11.640095 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:11.640473 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:11.641350 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:11.642820 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:11.643885 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:11.644611 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:11.644797 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:11.647592 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:11.647598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:11.647612 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:11.647642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:11.649917 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:11.650427 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:11.650433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:11.650439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:11.650593 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:11.650906 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:11.650910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:11.650912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:11.651478 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:11.651682 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:11.652597 (+ 915us) server_negotiation.cc:299] Negotiation successful
0619 16:47:11.652933 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260619 16:47:11.655944 14312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:11.661440 14312 catalog_manager.cc:1347] Generated new cluster ID: 78b35cca99204f4abbd786a124b3a344
I20260619 16:47:11.661515 14312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:11.675930 14312 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:11.677237 14312 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:11.696959 14312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6: Generated new TSK 0
I20260619 16:47:11.697841 14312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:47:11.725632 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35395
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:11.841449 14324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:11.841710 14324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:11.841754 14324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:11.845803 14324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:11.845867 14324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:11.845927 14324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:47:11.851286 14324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:11.852375 14324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:11.853318 14324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:11.862733 14330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:11.862728 14329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:11.862864 14324 server_base.cc:1034] running on GCE node
W20260619 16:47:11.862728 14332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:11.863435 14324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:11.864143 14324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:11.865316 14324 hybrid_clock.cc:648] HybridClock initialized: now 1781887631865283 us; error 46 us; skew 500 ppm
Jun 19 16:47:11 dist-test-slave-cgwx krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:11.868253 14324 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:47:11.869076 14324 webserver.cc:460] Webserver started at http://127.3.233.129:36033/ using document root <none> and password file <none>
I20260619 16:47:11.869619 14324 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:11.869680 14324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:11.869841 14324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:11.872126 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "e28f21d63ec446c4b4d7091e60dd4a96"
format_stamp: "Formatted at 2026-06-19 16:47:11 on dist-test-slave-cgwx"
server_key: "ca6302a8a6d91553ced0d66ca1d5956d"
server_key_iv: "2c921a476358e3edbc92187b0b87701b"
server_key_version: "encryptionkey@0"
I20260619 16:47:11.872641 14324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "e28f21d63ec446c4b4d7091e60dd4a96"
format_stamp: "Formatted at 2026-06-19 16:47:11 on dist-test-slave-cgwx"
server_key: "ca6302a8a6d91553ced0d66ca1d5956d"
server_key_iv: "2c921a476358e3edbc92187b0b87701b"
server_key_version: "encryptionkey@0"
I20260619 16:47:11.876259 14324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:47:11.878609 14339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:11.879870 14324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:47:11.879961 14324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "e28f21d63ec446c4b4d7091e60dd4a96"
format_stamp: "Formatted at 2026-06-19 16:47:11 on dist-test-slave-cgwx"
server_key: "ca6302a8a6d91553ced0d66ca1d5956d"
server_key_iv: "2c921a476358e3edbc92187b0b87701b"
server_key_version: "encryptionkey@0"
I20260619 16:47:11.880046 14324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:11.908660 14324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:11.911981 14324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:11.912137 14324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:11.912669 14324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:11.913435 14324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:11.913492 14324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:11.913534 14324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:11.913550 14324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:11.922891 14324 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:34331
I20260619 16:47:11.922974 14452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:34331 every 8 connection(s)
I20260619 16:47:11.923869 14324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:11.932456 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 14324
I20260619 16:47:11.932583 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:11.932879 4006 external_mini_cluster.cc:1428] Setting key e04928828cf33f79e4fafc468bffbf47
Jun 19 16:47:11 dist-test-slave-cgwx krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:11.938978 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:11.925810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:59109 (local address 127.3.233.190:35395)
0619 16:47:11.925977 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:11.925981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:11.926818 (+ 837us) server_negotiation.cc:407] Connection header received
0619 16:47:11.928247 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:11.928250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:11.928305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:11.928390 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:11.929640 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:11.930178 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:11.931350 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:11.931594 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:11.931708 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:11.934682 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:11.934685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:11.934687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:11.934711 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:11.936640 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:11.937132 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:11.937136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:11.937140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:11.937242 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:11.937565 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:11.937573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:11.937578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:11.937850 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:11.937974 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:11.938517 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 16:47:11.938669 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:47:11.939754 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:11.926200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35395 (local address 127.3.233.129:59109)
0619 16:47:11.926661 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:11.926704 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:11.928033 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:11.928628 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:11.928635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:11.928998 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:11.929474 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:11.929487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:11.930303 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:11.930306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:11.931222 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:11.931231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:11.931747 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:11.931750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:11.931917 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:11.932677 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:11.932701 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:11.934519 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:11.936756 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:11.936761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:11.936773 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:11.937030 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:11.937341 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:11.937343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:11.937344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:11.937470 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:11.938008 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:11.938014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:11.938252 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 16:47:11.938427 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 16:47:11.938772 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260619 16:47:11.941059 14453 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35395
I20260619 16:47:11.941298 14453 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:11.941795 14453 heartbeater.cc:511] Master 127.3.233.190:35395 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:47:11.943392 14262 ts_manager.cc:194] Registered new tserver with Master: e28f21d63ec446c4b4d7091e60dd4a96 (127.3.233.129:34331)
I20260619 16:47:11.944537 14262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:59109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:47:11.962534 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35395
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:12.075951 14458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:12.076251 14458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:12.076345 14458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:12.080448 14458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:12.080539 14458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:12.080623 14458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:47:12.086028 14458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:12.087178 14458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:12.088198 14458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:12.097821 14464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:12.097816 14463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:12.097816 14466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:12.098717 14458 server_base.cc:1034] running on GCE node
I20260619 16:47:12.099197 14458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:12.099828 14458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:12.101022 14458 hybrid_clock.cc:648] HybridClock initialized: now 1781887632101006 us; error 27 us; skew 500 ppm
Jun 19 16:47:12 dist-test-slave-cgwx krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:12.104048 14458 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:47:12.104913 14458 webserver.cc:460] Webserver started at http://127.3.233.130:44633/ using document root <none> and password file <none>
I20260619 16:47:12.105479 14458 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:12.105537 14458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:12.105754 14458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:12.107947 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "a6e9607c154243049be0556a5eba2bce"
format_stamp: "Formatted at 2026-06-19 16:47:12 on dist-test-slave-cgwx"
server_key: "cced001af923a6e541b23f0665e73a8e"
server_key_iv: "80426506b513014dbc7eddc976b624fc"
server_key_version: "encryptionkey@0"
I20260619 16:47:12.108501 14458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "a6e9607c154243049be0556a5eba2bce"
format_stamp: "Formatted at 2026-06-19 16:47:12 on dist-test-slave-cgwx"
server_key: "cced001af923a6e541b23f0665e73a8e"
server_key_iv: "80426506b513014dbc7eddc976b624fc"
server_key_version: "encryptionkey@0"
I20260619 16:47:12.112213 14458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:47:12.114675 14473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:12.116070 14458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:47:12.116191 14458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "a6e9607c154243049be0556a5eba2bce"
format_stamp: "Formatted at 2026-06-19 16:47:12 on dist-test-slave-cgwx"
server_key: "cced001af923a6e541b23f0665e73a8e"
server_key_iv: "80426506b513014dbc7eddc976b624fc"
server_key_version: "encryptionkey@0"
I20260619 16:47:12.116299 14458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:12.135825 14458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:12.139101 14458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:12.139303 14458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:12.139887 14458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:12.140705 14458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:12.140751 14458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:12.140820 14458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:12.140861 14458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:12.149333 14458 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:43455
I20260619 16:47:12.149412 14586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:43455 every 8 connection(s)
I20260619 16:47:12.150260 14458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:12.159198 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 14458
I20260619 16:47:12.159292 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:12.159600 4006 external_mini_cluster.cc:1428] Setting key e6c72a30d3098ccf6b98152c4fcd10a4
Jun 19 16:47:12 dist-test-slave-cgwx krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:12.165364 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.152277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:42909 (local address 127.3.233.190:35395)
0619 16:47:12.152423 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.152427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.153439 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 16:47:12.154812 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.154816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.154870 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.154978 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:12.156173 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.156707 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.157867 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.158111 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.158223 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.160869 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:12.160874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:12.160876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:12.160904 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:12.162703 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:12.163573 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:12.163576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:12.163577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:12.163654 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:12.164062 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:12.164064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:12.164065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:12.164256 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:12.164362 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.164964 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.165115 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:47:12.166169 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.152823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35395 (local address 127.3.233.130:42909)
0619 16:47:12.153291 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.153334 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:12.154604 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:12.155155 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:12.155162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:12.155559 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:12.156033 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.156046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.156836 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.156845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.157738 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.157747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.158262 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.158266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.158437 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.159057 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:12.159072 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:12.160690 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:12.162890 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:12.162899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:12.162917 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:12.163465 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:12.163749 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:12.163753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:12.163757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:12.163950 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:12.164448 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:12.164454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:12.164706 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 16:47:12.164893 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 16:47:12.165170 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260619 16:47:12.167533 14587 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35395
I20260619 16:47:12.167791 14587 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:12.168306 14587 heartbeater.cc:511] Master 127.3.233.190:35395 requested a full tablet report, sending...
I20260619 16:47:12.169423 14262 ts_manager.cc:194] Registered new tserver with Master: a6e9607c154243049be0556a5eba2bce (127.3.233.130:43455)
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:47:12.170117 14262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:42909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:47:12.187691 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35395
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:12.302712 14592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:12.303040 14592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:12.303092 14592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:12.307415 14592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:12.307482 14592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:12.307542 14592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:47:12.313016 14592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:12.314124 14592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:12.315263 14592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:12.325297 14600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:12.325289 14598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:12.325289 14597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:12.325698 14592 server_base.cc:1034] running on GCE node
I20260619 16:47:12.326151 14592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:12.326825 14592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:12.328027 14592 hybrid_clock.cc:648] HybridClock initialized: now 1781887632328011 us; error 32 us; skew 500 ppm
Jun 19 16:47:12 dist-test-slave-cgwx krb5kdc[14217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:12.331103 14592 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:47:12.332012 14592 webserver.cc:460] Webserver started at http://127.3.233.131:39295/ using document root <none> and password file <none>
I20260619 16:47:12.332551 14592 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:12.332610 14592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:12.332779 14592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:47:12.334827 14592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7"
format_stamp: "Formatted at 2026-06-19 16:47:12 on dist-test-slave-cgwx"
server_key: "37113467ac1a5e6d23e8ec869c100a1e"
server_key_iv: "0c174edeb1e5a42789df8fb872ff799e"
server_key_version: "encryptionkey@0"
I20260619 16:47:12.335426 14592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7"
format_stamp: "Formatted at 2026-06-19 16:47:12 on dist-test-slave-cgwx"
server_key: "37113467ac1a5e6d23e8ec869c100a1e"
server_key_iv: "0c174edeb1e5a42789df8fb872ff799e"
server_key_version: "encryptionkey@0"
I20260619 16:47:12.339123 14592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:47:12.341694 14607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:12.343360 14592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:47:12.343530 14592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7"
format_stamp: "Formatted at 2026-06-19 16:47:12 on dist-test-slave-cgwx"
server_key: "37113467ac1a5e6d23e8ec869c100a1e"
server_key_iv: "0c174edeb1e5a42789df8fb872ff799e"
server_key_version: "encryptionkey@0"
I20260619 16:47:12.343657 14592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:12.360143 14592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:12.363502 14592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:12.363719 14592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:12.364368 14592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:12.365190 14592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:12.365262 14592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:12.365334 14592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:12.365375 14592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:12.374159 14592 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:44065
I20260619 16:47:12.374184 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:44065 every 8 connection(s)
I20260619 16:47:12.375095 14592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
Jun 19 16:47:12 dist-test-slave-cgwx krb5kdc[14217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:12.384912 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 14592
I20260619 16:47:12.385051 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:12.385334 4006 external_mini_cluster.cc:1428] Setting key 1d3b1e4d8630744709c2c6acb63a2034
I20260619 16:47:12.389171 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.376875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:37905 (local address 127.3.233.190:35395)
0619 16:47:12.377051 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.377054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.377882 (+ 828us) server_negotiation.cc:407] Connection header received
0619 16:47:12.379456 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.379459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.379521 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.379621 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:12.380724 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.381244 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.382390 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.382606 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.382716 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.385491 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:12.385495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:12.385498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:12.385564 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:12.387167 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:12.387748 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:12.387751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:12.387753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:12.387844 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:12.388243 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:12.388245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:12.388246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:12.388417 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:12.388492 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.388963 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.389058 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:47:12.389983 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.377245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35395 (local address 127.3.233.131:37905)
0619 16:47:12.377695 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.377736 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:12.379219 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:12.379786 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:12.379792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:12.380134 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:12.380585 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.380597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.381383 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.381392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.382257 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.382265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.382761 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.382765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.382981 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.383590 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:12.383607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:12.385301 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:12.387274 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:12.387280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:12.387294 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:12.387647 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:12.387964 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:12.387974 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:12.387976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:12.388116 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:12.388511 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:12.388516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:12.388740 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 16:47:12.388920 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 16:47:12.389169 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260619 16:47:12.391163 14721 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35395
I20260619 16:47:12.391402 14721 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:12.391889 14721 heartbeater.cc:511] Master 127.3.233.190:35395 requested a full tablet report, sending...
I20260619 16:47:12.392853 14262 ts_manager.cc:194] Registered new tserver with Master: fbe40de76f0a43b3a3e89a2e8b84adb7 (127.3.233.131:44065)
I20260619 16:47:12.393390 14262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:37905
I20260619 16:47:12.399354 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:47:12.408562 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.401246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.3.233.190:35395)
0619 16:47:12.401369 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.401372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.401498 (+ 126us) server_negotiation.cc:407] Connection header received
0619 16:47:12.401589 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.401591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.401627 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.401696 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:12.402306 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.402886 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.403835 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.404062 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.404143 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.405443 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:12.405446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:12.405448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:12.405469 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:12.406862 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:12.407280 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:12.407283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:12.407284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:12.407416 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:12.407682 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:12.407685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:12.407686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:12.408001 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:12.408112 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.408303 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.408447 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:47:12.411548 14262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:47:12.414875 14262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:47:12.429971 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.424400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.3.233.129:34331)
0619 16:47:12.424892 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.424907 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.425379 (+ 472us) server_negotiation.cc:407] Connection header received
0619 16:47:12.425509 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.425515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.425722 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.425900 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:12.426258 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.427522 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.428725 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.429266 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.429401 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.429539 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.429716 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.429809 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":96,"threads_started":1}
I20260619 16:47:12.429976 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.423970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:43455 (local address 127.0.0.1:34560)
0619 16:47:12.424926 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.424979 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:12.425102 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:12.425863 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:12.425869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:12.425891 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:12.426033 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.426040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.427168 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.427173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.428342 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.428351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.429579 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.429583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.429673 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.429694 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:47:12.429758 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:47:12.429825 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":80,"threads_started":1}
I20260619 16:47:12.429978 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.424239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34331 (local address 127.0.0.1:37746)
0619 16:47:12.425280 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.425294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:12.425396 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:12.426037 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:12.426043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:12.426062 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:12.426159 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.426165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.427679 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.427682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.428598 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.428607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.429390 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.429394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.429503 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.429538 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:47:12.429688 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 16:47:12.429810 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":60,"thread_start_us":78,"threads_started":1}
I20260619 16:47:12.430692 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.424126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.3.233.130:43455)
0619 16:47:12.425333 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.425344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.425367 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:47:12.425493 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.425498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.425692 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.425851 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:12.426157 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.427042 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.428459 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.429282 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.429380 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.429520 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.430299 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.430396 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1090,"thread_start_us":134,"threads_started":1}
I20260619 16:47:12.431237 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.424519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.3.233.131:44065)
0619 16:47:12.425130 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.425142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.425682 (+ 540us) server_negotiation.cc:407] Connection header received
0619 16:47:12.425765 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.425772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.425949 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.426083 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:12.426390 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.427539 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.429592 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.430406 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.430502 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.430639 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.431002 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.431101 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":92,"threads_started":1}
I20260619 16:47:12.431684 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.423978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:44065 (local address 127.0.0.1:59452)
0619 16:47:12.425577 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.425590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:12.425667 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:12.426170 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:12.426173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:12.426190 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:12.426270 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.426275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.427679 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.427682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.429452 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.429461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.430705 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.430712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.430807 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.430833 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:47:12.430905 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:47:12.431558 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"mutex_wait_us":1216,"thread_start_us":97,"threads_started":1}
I20260619 16:47:12.433513 14387 tablet_service.cc:1467] Processing CreateTablet for tablet 4974871c9c7c450697d91b791b367238 (DEFAULT_TABLE table=test-table [id=fa0b85d887114963b7b972a0c8838b32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:12.433513 14521 tablet_service.cc:1467] Processing CreateTablet for tablet 4974871c9c7c450697d91b791b367238 (DEFAULT_TABLE table=test-table [id=fa0b85d887114963b7b972a0c8838b32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:12.433902 14655 tablet_service.cc:1467] Processing CreateTablet for tablet 4974871c9c7c450697d91b791b367238 (DEFAULT_TABLE table=test-table [id=fa0b85d887114963b7b972a0c8838b32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:12.434634 14521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4974871c9c7c450697d91b791b367238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:12.434634 14387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4974871c9c7c450697d91b791b367238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:12.434942 14655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4974871c9c7c450697d91b791b367238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:12.441059 14738 tablet_bootstrap.cc:492] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Bootstrap starting.
I20260619 16:47:12.442349 14739 tablet_bootstrap.cc:492] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96: Bootstrap starting.
I20260619 16:47:12.443270 14738 tablet_bootstrap.cc:654] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:12.443255 14740 tablet_bootstrap.cc:492] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Bootstrap starting.
I20260619 16:47:12.444375 14739 tablet_bootstrap.cc:654] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:12.444375 14738 log.cc:824] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:12.445231 14739 log.cc:824] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:12.445696 14740 tablet_bootstrap.cc:654] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:12.446430 14738 tablet_bootstrap.cc:492] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: No bootstrap required, opened a new log
I20260619 16:47:12.446475 14740 log.cc:824] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:12.446666 14738 ts_tablet_manager.cc:1397] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:47:12.447301 14739 tablet_bootstrap.cc:492] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96: No bootstrap required, opened a new log
I20260619 16:47:12.447479 14739 ts_tablet_manager.cc:1397] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 16:47:12.448415 14740 tablet_bootstrap.cc:492] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: No bootstrap required, opened a new log
I20260619 16:47:12.448590 14740 ts_tablet_manager.cc:1397] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:47:12.450137 14738 raft_consensus.cc:348] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.450479 14738 raft_consensus.cc:374] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:12.450557 14738 raft_consensus.cc:729] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe40de76f0a43b3a3e89a2e8b84adb7, State: Initialized, Role: FOLLOWER
I20260619 16:47:12.451047 14738 consensus_queue.cc:260] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.451710 14721 heartbeater.cc:503] Master 127.3.233.190:35395 was elected leader, sending a full tablet report...
I20260619 16:47:12.451642 14739 raft_consensus.cc:348] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.451874 14739 raft_consensus.cc:374] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:12.451933 14739 raft_consensus.cc:729] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28f21d63ec446c4b4d7091e60dd4a96, State: Initialized, Role: FOLLOWER
I20260619 16:47:12.452091 14738 ts_tablet_manager.cc:1428] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 16:47:12.452342 14739 consensus_queue.cc:260] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.452415 14740 raft_consensus.cc:348] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.452653 14740 raft_consensus.cc:374] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:12.452700 14740 raft_consensus.cc:729] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e9607c154243049be0556a5eba2bce, State: Initialized, Role: FOLLOWER
I20260619 16:47:12.452960 14453 heartbeater.cc:503] Master 127.3.233.190:35395 was elected leader, sending a full tablet report...
I20260619 16:47:12.453135 14740 consensus_queue.cc:260] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.453298 14739 ts_tablet_manager.cc:1428] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 16:47:12.453821 14587 heartbeater.cc:503] Master 127.3.233.190:35395 was elected leader, sending a full tablet report...
I20260619 16:47:12.454006 14740 ts_tablet_manager.cc:1428] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:47:12.550595 14746 raft_consensus.cc:482] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:12.550832 14746 raft_consensus.cc:504] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.552244 14746 leader_election.cc:288] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe40de76f0a43b3a3e89a2e8b84adb7 (127.3.233.131:44065), e28f21d63ec446c4b4d7091e60dd4a96 (127.3.233.129:34331)
I20260619 16:47:12.557096 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.552739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:50797 (local address 127.3.233.129:34331)
0619 16:47:12.552875 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.552879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.553026 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:47:12.553093 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.553096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.553137 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.553228 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:12.553465 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.554186 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.555739 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.556363 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.556473 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.556524 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.556901 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.556970 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:47:12.557096 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.552560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:52023 (local address 127.3.233.131:44065)
0619 16:47:12.552711 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.552716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.552732 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:47:12.552802 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.552807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.552857 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.552935 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:12.553242 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.554046 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.555739 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.556343 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.556473 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.556561 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.556900 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.556970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:47:12.557099 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.552428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:44065 (local address 127.3.233.130:52023)
0619 16:47:12.552539 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.552558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:12.552652 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:12.552958 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:12.552963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:12.552986 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:12.553134 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.553139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.554153 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.554156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.555584 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.555595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.556492 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.556496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.556719 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.556772 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 16:47:12.556876 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:47:12.556933 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":218}
I20260619 16:47:12.557107 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.552540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34331 (local address 127.3.233.130:50797)
0619 16:47:12.552917 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.552932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:12.553023 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:12.553237 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:12.553240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:12.553257 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:12.553358 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.553362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.554278 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.554281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.555590 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:12.555595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.556492 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.556496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:12.556702 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.556776 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 16:47:12.556856 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:47:12.556936 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":92,"thread_start_us":131,"threads_started":1}
I20260619 16:47:12.557829 14675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974871c9c7c450697d91b791b367238" candidate_uuid: "a6e9607c154243049be0556a5eba2bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" is_pre_election: true
I20260619 16:47:12.557919 14407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974871c9c7c450697d91b791b367238" candidate_uuid: "a6e9607c154243049be0556a5eba2bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" is_pre_election: true
I20260619 16:47:12.558169 14675 raft_consensus.cc:2454] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e9607c154243049be0556a5eba2bce in term 0.
I20260619 16:47:12.558287 14407 raft_consensus.cc:2454] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e9607c154243049be0556a5eba2bce in term 0.
I20260619 16:47:12.558713 14474 leader_election.cc:302] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a6e9607c154243049be0556a5eba2bce, e28f21d63ec446c4b4d7091e60dd4a96, fbe40de76f0a43b3a3e89a2e8b84adb7; no voters:
I20260619 16:47:12.558990 14746 raft_consensus.cc:2781] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:47:12.559069 14746 raft_consensus.cc:482] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:47:12.559110 14746 raft_consensus.cc:3037] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:12.560150 14746 raft_consensus.cc:504] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.560491 14746 leader_election.cc:288] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [CANDIDATE]: Term 1 election: Requested vote from peers fbe40de76f0a43b3a3e89a2e8b84adb7 (127.3.233.131:44065), e28f21d63ec446c4b4d7091e60dd4a96 (127.3.233.129:34331)
I20260619 16:47:12.560803 14675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974871c9c7c450697d91b791b367238" candidate_uuid: "a6e9607c154243049be0556a5eba2bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7"
I20260619 16:47:12.560894 14407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974871c9c7c450697d91b791b367238" candidate_uuid: "a6e9607c154243049be0556a5eba2bce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28f21d63ec446c4b4d7091e60dd4a96"
I20260619 16:47:12.560909 14675 raft_consensus.cc:3037] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:12.561022 14407 raft_consensus.cc:3037] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:12.561935 14675 raft_consensus.cc:2454] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e9607c154243049be0556a5eba2bce in term 1.
I20260619 16:47:12.562148 14407 raft_consensus.cc:2454] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e9607c154243049be0556a5eba2bce in term 1.
I20260619 16:47:12.562456 14477 leader_election.cc:302] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e9607c154243049be0556a5eba2bce, e28f21d63ec446c4b4d7091e60dd4a96; no voters:
I20260619 16:47:12.562659 14746 raft_consensus.cc:2781] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:12.562901 14746 raft_consensus.cc:686] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 1 LEADER]: Becoming Leader. State: Replica: a6e9607c154243049be0556a5eba2bce, State: Running, Role: LEADER
I20260619 16:47:12.563264 14746 consensus_queue.cc:237] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } }
I20260619 16:47:12.566274 14260 catalog_manager.cc:5495] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e9607c154243049be0556a5eba2bce (127.3.233.130). New cstate: current_term: 1 leader_uuid: "a6e9607c154243049be0556a5eba2bce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e9607c154243049be0556a5eba2bce" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 43455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 } health_report { overall_health: UNKNOWN } } }
I20260619 16:47:12.583168 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.579721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.3.233.130:43455)
0619 16:47:12.579837 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:12.579840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:12.579854 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:47:12.579942 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:12.579944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:12.579994 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:12.580092 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:12.580337 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.580848 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.581781 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:12.581986 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:12.582070 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:12.582340 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:12.582740 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:47:12.582874 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:12.582995 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 16:47:12.583046 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:47:12.588863 14407 raft_consensus.cc:1261] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 1 FOLLOWER]: Refusing update from remote peer a6e9607c154243049be0556a5eba2bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:12.588867 14675 raft_consensus.cc:1261] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 1 FOLLOWER]: Refusing update from remote peer a6e9607c154243049be0556a5eba2bce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:12.589550 14748 consensus_queue.cc:1035] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe40de76f0a43b3a3e89a2e8b84adb7" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 44065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:12.589766 14746 consensus_queue.cc:1035] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28f21d63ec446c4b4d7091e60dd4a96" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:12.598891 14753 mvcc.cc:205] Tried to move back new op lower bound from 7298611743077339136 to 7298611742981767168. Current Snapshot: MvccSnapshot[applied={T|T < 7298611743077339136 or (T in {7298611743077339136})}]
I20260619 16:47:12.606575 14260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:12.606787 14260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:12.609558 14260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ceb799cdc52a477c8df6c1a85ddf1ab6: Sending DeleteTablet for 3 replicas of tablet 4974871c9c7c450697d91b791b367238
I20260619 16:47:12.610494 14655 tablet_service.cc:1514] Processing DeleteTablet for tablet 4974871c9c7c450697d91b791b367238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:12 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:59452
I20260619 16:47:12.610566 14387 tablet_service.cc:1514] Processing DeleteTablet for tablet 4974871c9c7c450697d91b791b367238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:12 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:37746
I20260619 16:47:12.610621 14521 tablet_service.cc:1514] Processing DeleteTablet for tablet 4974871c9c7c450697d91b791b367238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:12 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:34560
I20260619 16:47:12.611106 14763 tablet_replica.cc:330] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96: stopping tablet replica
I20260619 16:47:12.611196 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 14324
I20260619 16:47:12.611334 14761 tablet_replica.cc:330] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: stopping tablet replica
I20260619 16:47:12.611511 14763 raft_consensus.cc:2229] T 4974871c9c7c450697d91b791b367238 P e28f21d63ec446c4b4d7091e60dd4a96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:12.611595 14761 raft_consensus.cc:2229] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:12.611848 14762 tablet_replica.cc:330] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: stopping tablet replica
I20260619 16:47:12.612087 14762 raft_consensus.cc:2229] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:47:12.612452 14762 raft_consensus.cc:2258] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:12.612882 14761 raft_consensus.cc:2258] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:12.614118 14762 ts_tablet_manager.cc:1905] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:12.614395 14761 ts_tablet_manager.cc:1905] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:12.617311 14761 ts_tablet_manager.cc:1918] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:12.617429 14761 log.cc:1196] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/4974871c9c7c450697d91b791b367238
I20260619 16:47:12.617784 14761 ts_tablet_manager.cc:1939] T 4974871c9c7c450697d91b791b367238 P fbe40de76f0a43b3a3e89a2e8b84adb7: Deleting consensus metadata
I20260619 16:47:12.617877 14762 ts_tablet_manager.cc:1918] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:12.618016 14762 log.cc:1196] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/4974871c9c7c450697d91b791b367238
I20260619 16:47:12.618366 14762 ts_tablet_manager.cc:1939] T 4974871c9c7c450697d91b791b367238 P a6e9607c154243049be0556a5eba2bce: Deleting consensus metadata
I20260619 16:47:12.619266 14246 catalog_manager.cc:4841] TS fbe40de76f0a43b3a3e89a2e8b84adb7 (127.3.233.131:44065): tablet 4974871c9c7c450697d91b791b367238 (table test-table [id=fa0b85d887114963b7b972a0c8838b32]) successfully deleted
W20260619 16:47:12.621424 14249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:34331 (error 108)
I20260619 16:47:12.621680 14247 catalog_manager.cc:4841] TS a6e9607c154243049be0556a5eba2bce (127.3.233.130:43455): tablet 4974871c9c7c450697d91b791b367238 (table test-table [id=fa0b85d887114963b7b972a0c8838b32]) successfully deleted
I20260619 16:47:12.621986 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 14458
I20260619 16:47:12.623703 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:12.623391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34331 (local address 127.0.0.1:33282)
0619 16:47:12.623570 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:12.623631 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:34331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:47:12.623898 14249 catalog_manager.cc:4584] TS e28f21d63ec446c4b4d7091e60dd4a96 (127.3.233.129:34331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4974871c9c7c450697d91b791b367238: Network error: Client connection negotiation failed: client connection to 127.3.233.129:34331: connect: Connection refused (error 111)
I20260619 16:47:12.628423 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 14592
I20260619 16:47:12.635797 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 14230
2026-06-19T16:47:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2362 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:47:12 dist-test-slave-cgwx krb5kdc[14767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:47:12 dist-test-slave-cgwx krb5kdc[14767](info): set up 2 sockets
Jun 19 16:47:12 dist-test-slave-cgwx krb5kdc[14767](info): commencing operation
krb5kdc: starting...
W20260619 16:47:13.700083 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:47:13 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:47:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/rangeradmin_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/rangeradmin_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/rangerlookup_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/rangerlookup_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
I20260619 16:47:13.874634 4006 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfHMdDR/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/postgres -l logfile start
2026-06-19 16:47:16.436 UTC [14797] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:47:16.436 UTC [14797] LOG: listening on IPv4 address "127.3.233.148", port 41833
2026-06-19 16:47:16.439 UTC [14797] LOG: listening on Unix socket "/tmp/.s.PGSQL.41833"
2026-06-19 16:47:16.461 UTC [14800] LOG: database system was shut down at 2026-06-19 16:47:14 UTC
2026-06-19 16:47:16.465 UTC [14797] LOG: database system is ready to accept connections
I20260619 16:47:17.424392 4006 mini_postgres.cc:96] Postgres bound to 41833
2026-06-19 16:47:17.431 UTC [14808] FATAL: database "slave" does not exist
127.3.233.148:41833 - accepting connections
I20260619 16:47:17.431840 4006 mini_ranger.cc:161] Starting Ranger...
I20260619 16:47:17.448103 4006 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:47:17.968137 4006 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:47:17.968223 4006 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-admin
2026-06-19 16:47:18,259 [I] DB FLAVOR :POSTGRES
2026-06-19 16:47:18,259 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:47:18,259 [I] Checking connection..
2026-06-19 16:47:18,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:47:18,508 [I] Checking connection passed.
2026-06-19 16:47:18,508 [I] --------- Verifying version history table ---------
2026-06-19 16:47:18,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:47:18,746 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:47:18,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:47:18,993 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:47:18,993 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:47:18,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:47:19.252 UTC [14872] WARNING: there is no transaction in progress
2026-06-19 16:47:19,258 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:47:19,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:47:19,503 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:47:19,503 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:47:19,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:47:19,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:47:20,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cgwx.c.gcp-upstream.internal','N') ;"
2026-06-19 16:47:20,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:47:20,511 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:47:20,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:47:20,763 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:47:20,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:47:21,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:47:21,251 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:47:21,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:47:22.242 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.259 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.276 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.292 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.308 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.326 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.385 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.392 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.406 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.416 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.427 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.441 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.452 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.459 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.466 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:22.472 UTC [14980] WARNING: there is no transaction in progress
2026-06-19 16:47:23,148 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:47:23,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:47:23,391 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:47:23,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:47:23,680 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:47:23,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:47:23,961 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:47:23,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:47:24,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:47:24,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cgwx.c.gcp-upstream.internal';"
2026-06-19 16:47:24,735 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:47:24,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:47:25,018 [I] DB_PATCHES have already been applied
I20260619 16:47:25.023897 4006 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfHMdDR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:47:25.024010 4006 mini_ranger.cc:193] Using host: 127.3.233.148
I20260619 16:47:25.027902 4006 mini_ranger.cc:227] Ranger admin URL: http://127.3.233.148:51541
Jun 19, 2026 4:47:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:47:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:47:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:47:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:47:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:47:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:47:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:47:26.145056 4006 mini_ranger.cc:160] Time spent starting Ranger: real 8.713s user 0.007s sys 0.004s
Jun 19, 2026 4:47:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:47:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 4:47:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:47:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:47:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:47:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:47:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:47:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:47:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:47:48.522051 4006 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/rangerkms_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/rangerkms_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.233.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.233.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:47:48 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:47:48.611810 4006 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:47:48.622769 4006 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:47:49.475109 4006 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:47:49.475217 4006 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms
I20260619 16:47:49.475314 4006 mini_ranger_kms.cc:227] Using postgres at 127.3.233.148:41833
2026-06-19 16:47:49,687 [I] DB FLAVOR :POSTGRES
2026-06-19 16:47:49,687 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:47:49,687 [I] Checking connection
2026-06-19 16:47:49,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:47:49,949 [I] connection success
2026-06-19 16:47:49,950 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:47:49,950 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:47:49,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:47:50,220 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:47:50,220 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:47:50,220 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:47:50,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:41833/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:47:50,509 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:47:51.002103 4006 mini_ranger_kms.cc:310] Created kms service
I20260619 16:47:51.181313 4006 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:47:51.225324 4006 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:47:51 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887668, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:47:51.609232 4006 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:47:51.609403 4006 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:47:51.609431 4006 mini_ranger_kms.cc:240] Using host: 127.3.233.148
I20260619 16:47:51.613706 4006 mini_ranger_kms.cc:276] Ranger KMS PID: 15160
I20260619 16:47:51.613867 4006 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.233.148:54411
Jun 19, 2026 4:47:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:47:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:47:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:47:54.018999 4006 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.407s user 0.005s sys 0.190s
I20260619 16:47:54.019148 4006 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:47:54.019268 4006 mini_ranger_kms.cc:401] 127.3.233.148:54411/kms/v1/keys
Jun 19 16:47:54 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.233.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:47:54 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.233.148@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
Jun 19 16:47:55 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887674, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.233.148@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
Jun 19 16:47:57 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:47:57.303651 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34499
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34499
--ranger_config_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:54411/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:57.428364 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:57.428668 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:57.428743 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:57.432794 15211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:47:57.432878 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:57.432914 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:57.432967 15211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:47:57.433003 15211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:47:57.438550 15211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.233.148:54411/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34499
--ranger_config_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:57.439735 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:57.440809 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:57.451042 15219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:57.451045 15217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:57.451043 15216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:57.451248 15211 server_base.cc:1034] running on GCE node
I20260619 16:47:57.452086 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:57.453313 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:57.454515 15211 hybrid_clock.cc:648] HybridClock initialized: now 1781887677454490 us; error 48 us; skew 500 ppm
Jun 19 16:47:57 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887677, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:57.457459 15211 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:47:57.458437 15211 webserver.cc:460] Webserver started at http://127.3.233.190:37335/ using document root <none> and password file <none>
I20260619 16:47:57.459064 15211 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:57.459125 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:57.459342 15211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:47:57 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887677, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:47:57.521715 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "7f6243b23e0d40208fc46ef16056fe9a"
format_stamp: "Formatted at 2026-06-19 16:47:57 on dist-test-slave-cgwx"
server_key: "fd970d0cef044b6d8310024704a40ec8"
server_key_iv: "8ee1835b7c8882423107092481868cd4"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:57.522354 15211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "7f6243b23e0d40208fc46ef16056fe9a"
format_stamp: "Formatted at 2026-06-19 16:47:57 on dist-test-slave-cgwx"
server_key: "fd970d0cef044b6d8310024704a40ec8"
server_key_iv: "8ee1835b7c8882423107092481868cd4"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:57.526404 15211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:47:57.544189 15226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:57.545673 15211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:47:57.545812 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "7f6243b23e0d40208fc46ef16056fe9a"
format_stamp: "Formatted at 2026-06-19 16:47:57 on dist-test-slave-cgwx"
server_key: "fd970d0cef044b6d8310024704a40ec8"
server_key_iv: "8ee1835b7c8882423107092481868cd4"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:57.545945 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:57.559242 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:57.562824 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:57.563057 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:57.571079 15211 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34499
I20260619 16:47:57.571111 15278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34499 every 8 connection(s)
I20260619 16:47:57.572067 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:47:57.574833 15279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:57.580773 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a: Bootstrap starting.
I20260619 16:47:57.582332 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 15211
I20260619 16:47:57.582461 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:47:57.582764 4006 external_mini_cluster.cc:1428] Setting key d7bd2726c52e6147a93a286d2e8e24e2
I20260619 16:47:57.583428 15279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:57.584429 15279 log.cc:824] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:57.586659 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a: No bootstrap required, opened a new log
I20260619 16:47:57.591045 15279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6243b23e0d40208fc46ef16056fe9a" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34499 } }
I20260619 16:47:57.591367 15279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:57.591446 15279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6243b23e0d40208fc46ef16056fe9a, State: Initialized, Role: FOLLOWER
I20260619 16:47:57.591951 15279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6243b23e0d40208fc46ef16056fe9a" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34499 } }
I20260619 16:47:57.592100 15279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:47:57.592188 15279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:47:57.592322 15279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:47:57 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:57.593578 15279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6243b23e0d40208fc46ef16056fe9a" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34499 } }
I20260619 16:47:57.593953 15279 leader_election.cc:302] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f6243b23e0d40208fc46ef16056fe9a; no voters:
I20260619 16:47:57.594338 15279 leader_election.cc:288] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:47:57.594451 15284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:57.594694 15284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [term 1 LEADER]: Becoming Leader. State: Replica: 7f6243b23e0d40208fc46ef16056fe9a, State: Running, Role: LEADER
I20260619 16:47:57.595362 15284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6243b23e0d40208fc46ef16056fe9a" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34499 } }
I20260619 16:47:57.595803 15279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:47:57.598209 15279 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:47:57.599606 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:57.583976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.3.233.190:34499)
0619 16:47:57.584461 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:57.584480 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:57.584523 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:47:57.585285 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:57.585310 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:57.585720 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:57.586077 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:57.586961 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.588516 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.589636 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:57.590374 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:57.590538 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:57.593658 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:57.593665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:57.593683 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:57.593715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:57.595871 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.596414 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.596420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.596426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.596555 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:57.596883 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:57.596887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:57.596888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:57.597423 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:57.597609 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:57.598355 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 16:47:57.598630 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":150,"threads_started":1}
I20260619 16:47:57.602774 15286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6243b23e0d40208fc46ef16056fe9a. Latest consensus state: current_term: 1 leader_uuid: "7f6243b23e0d40208fc46ef16056fe9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6243b23e0d40208fc46ef16056fe9a" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34499 } } }
I20260619 16:47:57.603155 15286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:57.603796 15285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f6243b23e0d40208fc46ef16056fe9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f6243b23e0d40208fc46ef16056fe9a" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34499 } } }
I20260619 16:47:57.603964 15285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a [sys.catalog]: This master's current role is: LEADER
I20260619 16:47:57.605114 15301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:47:57.608522 15301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:47:57.621035 15301 catalog_manager.cc:1347] Generated new cluster ID: 3ab0017a620945368a1aa6c89548dcbb
I20260619 16:47:57.621109 15301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:47:57.655076 15301 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:47:57.656594 15301 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:47:57.666236 15301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a: Generated new TSK 0
I20260619 16:47:57.667223 15301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:47:58.402045 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34499
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:47:58.547612 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:58.548079 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:58.548208 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:58.552826 15324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:58.552909 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:58.553026 15324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:47:58.558846 15324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.233.148:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:58.560060 15324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:58.561091 15324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:58.570752 15331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:58.570752 15334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:58.571318 15332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:58.571707 15324 server_base.cc:1034] running on GCE node
I20260619 16:47:58.572160 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:58.572922 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:58.576202 15324 hybrid_clock.cc:648] HybridClock initialized: now 1781887678576158 us; error 51 us; skew 500 ppm
Jun 19 16:47:58 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:58.584682 15324 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:47:58.585942 15324 webserver.cc:460] Webserver started at http://127.3.233.129:38123/ using document root <none> and password file <none>
I20260619 16:47:58.586639 15324 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:58.586722 15324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:58.586961 15324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:47:58 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:47:58.605803 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "6a1da4e559854c11980aaae7e5740e44"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-cgwx"
server_key: "bd5394b4dc09bf9ca4d3f4373ac7188f"
server_key_iv: "9197f879599c8bfa7c9449a2c78d4f1b"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:58.607487 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "6a1da4e559854c11980aaae7e5740e44"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-cgwx"
server_key: "bd5394b4dc09bf9ca4d3f4373ac7188f"
server_key_iv: "9197f879599c8bfa7c9449a2c78d4f1b"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:58.616516 15324 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260619 16:47:58.638723 15341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:58.640522 15324 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 16:47:58.640656 15324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "6a1da4e559854c11980aaae7e5740e44"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-cgwx"
server_key: "bd5394b4dc09bf9ca4d3f4373ac7188f"
server_key_iv: "9197f879599c8bfa7c9449a2c78d4f1b"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:58.640795 15324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:58.660027 15324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:58.663828 15324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:58.664106 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:58.664825 15324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:58.665890 15324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:58.665953 15324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:58.666008 15324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:58.666080 15324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:47:58.678264 15324 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:42979
I20260619 16:47:58.679613 15324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:47:58.681417 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 15324
I20260619 16:47:58.681553 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:47:58.681879 4006 external_mini_cluster.cc:1428] Setting key 9779be9ef62395b68ef9de1d10ed32a5
I20260619 16:47:58.682158 15456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:42979 every 8 connection(s)
Jun 19 16:47:58 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:58.701496 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:58.686479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:54205 (local address 127.3.233.190:34499)
0619 16:47:58.688733 (+ 2254us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:58.688738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:58.688752 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:47:58.688795 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:58.688798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:58.688845 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:58.688931 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:58.690444 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.691398 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.693174 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.693547 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.693645 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:58.696939 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:58.696944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:58.696946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:58.696974 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:58.698902 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:58.699577 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:58.699581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:58.699583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:58.699674 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:58.700118 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:58.700122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:58.700124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:58.700359 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:58.700431 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:58.701168 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 16:47:58.701318 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2160,"thread_start_us":59,"threads_started":1}
I20260619 16:47:58.703454 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:58.685387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34499 (local address 127.3.233.129:54205)
0619 16:47:58.687222 (+ 1835us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:58.687276 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:58.688297 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:58.689236 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:58.689245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:58.689597 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:58.690270 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:58.690289 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.691549 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.691553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:58.692986 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:58.692997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:58.693840 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:58.693844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:58.694044 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:58.694772 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:58.694793 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:58.696710 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:58.699072 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:58.699079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:58.699098 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:58.699453 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:58.699805 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:58.699808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:58.699810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:58.700002 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:58.700513 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:58.700519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:58.700879 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 16:47:58.701976 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0619 16:47:58.702333 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1611,"thread_start_us":141,"threads_started":1}
I20260619 16:47:58.705179 15457 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34499
I20260619 16:47:58.705421 15457 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:58.705997 15457 heartbeater.cc:511] Master 127.3.233.190:34499 requested a full tablet report, sending...
I20260619 16:47:58.707860 15243 ts_manager.cc:194] Registered new tserver with Master: 6a1da4e559854c11980aaae7e5740e44 (127.3.233.129:42979)
I20260619 16:47:58.709398 15243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:54205
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:47:58.733992 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34499
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:47:58 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:47:58.868701 15463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:58.868994 15463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:58.869092 15463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:58.873633 15463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:58.873734 15463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:58.873834 15463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:47:58.879632 15463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.233.148:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:58.880851 15463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:58.881937 15463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:58.893054 15469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:58.893114 15472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:58.893369 15463 server_base.cc:1034] running on GCE node
W20260619 16:47:58.893577 15470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:58.894080 15463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:58.894876 15463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:58.896095 15463 hybrid_clock.cc:648] HybridClock initialized: now 1781887678896054 us; error 56 us; skew 500 ppm
Jun 19 16:47:58 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:58.899925 15463 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:47:58.901062 15463 webserver.cc:460] Webserver started at http://127.3.233.130:46027/ using document root <none> and password file <none>
I20260619 16:47:58.901862 15463 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:58.901958 15463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:58.902192 15463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:47:58 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:47:58.919410 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "b63be1c223a2458bac2d87bb01715005"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-cgwx"
server_key: "e9b72bfedf6444001a3485c5ec4a9a13"
server_key_iv: "7e1c0582020cb47c8d36cabad5a832fa"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:58.920251 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "b63be1c223a2458bac2d87bb01715005"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-cgwx"
server_key: "e9b72bfedf6444001a3485c5ec4a9a13"
server_key_iv: "7e1c0582020cb47c8d36cabad5a832fa"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:58.927043 15463 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 16:47:58.962324 15479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:58.964257 15463 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.000s
I20260619 16:47:58.964450 15463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "b63be1c223a2458bac2d87bb01715005"
format_stamp: "Formatted at 2026-06-19 16:47:58 on dist-test-slave-cgwx"
server_key: "e9b72bfedf6444001a3485c5ec4a9a13"
server_key_iv: "7e1c0582020cb47c8d36cabad5a832fa"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:58.965029 15463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:58.998572 15463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:59.002568 15463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:59.002803 15463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:59.003497 15463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:59.004490 15463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:59.004601 15463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.004956 15463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:59.005038 15463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.016860 15463 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:33977
I20260619 16:47:59.018148 15463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:47:59.016868 15592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:33977 every 8 connection(s)
I20260619 16:47:59.023175 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 15463
I20260619 16:47:59.023322 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:47:59.023622 4006 external_mini_cluster.cc:1428] Setting key c39d01d4f54e6e2a301eafefc660b039
Jun 19 16:47:59 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:59.039525 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.021074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:48913 (local address 127.3.233.190:34499)
0619 16:47:59.021217 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.021223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.022646 (+ 1423us) server_negotiation.cc:407] Connection header received
0619 16:47:59.024930 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.024934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.024988 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.025041 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:59.026766 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.027636 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.031456 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.031797 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.031915 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.035026 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:59.035030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:59.035033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:59.035063 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:59.036795 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:59.037601 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:59.037605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:59.037607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:59.037699 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:59.038132 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:59.038135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:59.038136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:59.038377 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:59.038475 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.039225 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.039365 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:47:59.041172 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.021231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34499 (local address 127.3.233.130:48913)
0619 16:47:59.021799 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.021859 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.024637 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.025472 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.025481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.025869 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:59.026561 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.026582 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.027820 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.027824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.031287 (+ 3463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.031302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.031984 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.031991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.032243 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.032965 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:59.032988 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:59.034826 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:59.037028 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:59.037037 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:59.037060 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:59.037466 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:59.037803 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:59.037808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:59.037811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:59.038018 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:59.038497 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:59.038506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:59.038884 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.039141 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.039511 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":141,"threads_started":1}
I20260619 16:47:59.042506 15593 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34499
I20260619 16:47:59.042780 15593 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:59.043366 15593 heartbeater.cc:511] Master 127.3.233.190:34499 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:47:59.044486 15243 ts_manager.cc:194] Registered new tserver with Master: b63be1c223a2458bac2d87bb01715005 (127.3.233.130:33977)
I20260619 16:47:59.045064 15243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:48913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:47:59.074651 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:54411/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34499
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:35019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:47:59 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
W20260619 16:47:59.250419 15598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:47:59.251029 15598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:47:59.251145 15598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:47:59.257375 15598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:47:59.257557 15598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:47:59.257705 15598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:47:59.265806 15598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:35019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.233.148:54411/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:47:59.267490 15598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:47:59.268874 15598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:47:59.282276 15606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:47:59.282582 15604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:59.283902 15598 server_base.cc:1034] running on GCE node
W20260619 16:47:59.284236 15603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:47:59.284778 15598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:47:59.285570 15598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:47:59.286794 15598 hybrid_clock.cc:648] HybridClock initialized: now 1781887679286728 us; error 82 us; skew 500 ppm
Jun 19 16:47:59 dist-test-slave-cgwx krb5kdc[14767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:47:59.290462 15598 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:47:59.291591 15598 webserver.cc:460] Webserver started at http://127.3.233.131:36065/ using document root <none> and password file <none>
I20260619 16:47:59.292358 15598 fs_manager.cc:359] Metadata directory not provided
I20260619 16:47:59.292430 15598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:47:59.292649 15598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:47:59 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:47:59.350389 15598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "48f96fa2b53449bebd9b5a265aad1b48"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-cgwx"
server_key: "fb323dec4931a524279f9ddcf5937e15"
server_key_iv: "224baf4abe36173a2fc3652d2f3ad234"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:59.356463 15598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "48f96fa2b53449bebd9b5a265aad1b48"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-cgwx"
server_key: "fb323dec4931a524279f9ddcf5937e15"
server_key_iv: "224baf4abe36173a2fc3652d2f3ad234"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:59.365020 15598 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.004s
I20260619 16:47:59.396685 15613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.400497 15598 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260619 16:47:59.400779 15598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "48f96fa2b53449bebd9b5a265aad1b48"
format_stamp: "Formatted at 2026-06-19 16:47:59 on dist-test-slave-cgwx"
server_key: "fb323dec4931a524279f9ddcf5937e15"
server_key_iv: "224baf4abe36173a2fc3652d2f3ad234"
server_key_version: "kuduclusterkey@0"
I20260619 16:47:59.401027 15598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:47:59.424417 15598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:47:59.429775 15598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:47:59.430052 15598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:47:59.430946 15598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:47:59.432215 15598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:47:59.432337 15598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.432437 15598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:47:59.432497 15598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:47:59.445552 15598 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:40601
I20260619 16:47:59.447187 15598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:47:59.457818 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 15598
I20260619 16:47:59.457957 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:47:59.458289 4006 external_mini_cluster.cc:1428] Setting key d11817c6631b8f0e0db5b7f6dfb9543f
I20260619 16:47:59.461251 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:40601 every 8 connection(s)
Jun 19 16:47:59 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:47:59.480026 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.458728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:39201 (local address 127.3.233.190:34499)
0619 16:47:59.458984 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.458992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.462136 (+ 3144us) server_negotiation.cc:407] Connection header received
0619 16:47:59.463722 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.463726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.463791 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.463853 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:59.465756 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.466723 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.468755 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.469198 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.469289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.474836 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:59.474843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:59.474849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:59.474887 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:59.476943 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:59.477803 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:59.477809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:59.477814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:59.477914 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:59.478377 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:59.478383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:59.478388 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:59.478677 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:59.478769 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.479614 (+ 845us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.479773 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260619 16:47:59.482239 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.456384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34499 (local address 127.3.233.131:39201)
0619 16:47:59.461974 (+ 5590us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.462035 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.463432 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.464359 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.464372 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.464780 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:47:59.465566 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.465598 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.466898 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.466902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.468599 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.468614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.469530 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.469535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.469797 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.470596 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:47:59.470624 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:47:59.474614 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:47:59.477130 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:59.477139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:59.477165 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:59.477666 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:59.478032 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:47:59.478037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:47:59.478042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:47:59.478263 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:47:59.478845 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:47:59.478856 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:47:59.479282 (+ 426us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.480568 (+ 1286us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.481009 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5314,"thread_start_us":155,"threads_started":1}
I20260619 16:47:59.485492 15727 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34499
I20260619 16:47:59.485797 15727 heartbeater.cc:464] Registering TS with master...
I20260619 16:47:59.486467 15727 heartbeater.cc:511] Master 127.3.233.190:34499 requested a full tablet report, sending...
I20260619 16:47:59.488212 15243 ts_manager.cc:194] Registered new tserver with Master: 48f96fa2b53449bebd9b5a265aad1b48 (127.3.233.131:40601)
I20260619 16:47:59.489105 15243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:39201
I20260619 16:47:59.499992 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:47:59.513907 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.502634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.3.233.190:34499)
0619 16:47:59.502757 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.502762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.503420 (+ 658us) server_negotiation.cc:407] Connection header received
0619 16:47:59.503661 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.503665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.503727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.503807 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:47:59.505473 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.506306 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.507737 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.508074 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.508156 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.510105 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:47:59.510109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:47:59.510112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:47:59.510146 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:47:59.511939 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:59.512507 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:59.512511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:59.512513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:59.512604 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:47:59.512913 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:47:59.512916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:47:59.512918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:47:59.513140 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:47:59.513241 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.513595 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.513740 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:47:59.518357 15243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:47:59.523373 15243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:47:59.571079 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.536857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:33977 (local address 127.0.0.1:38546)
0619 16:47:59.548450 (+ 11593us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.548533 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.548720 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.554782 (+ 6062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.554789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.554812 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.554983 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.554992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.561029 (+ 6037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.561034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.562706 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.562724 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.570240 (+ 7516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.570247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.570450 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.570513 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.570645 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.570855 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11346,"thread_start_us":128,"threads_started":1}
I20260619 16:47:59.572315 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.542346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.3.233.129:42979)
0619 16:47:59.547819 (+ 5473us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.547830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.550323 (+ 2493us) server_negotiation.cc:407] Connection header received
0619 16:47:59.550425 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.550431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.550683 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.551025 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.552918 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.554285 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.565062 (+ 10777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.566402 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.566558 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.566787 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.572025 (+ 5238us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.572162 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5301,"thread_start_us":97,"threads_started":1}
I20260619 16:47:59.571883 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.542183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42979 (local address 127.0.0.1:56770)
0619 16:47:59.550081 (+ 7898us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.550097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.550209 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.552468 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.552479 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.552514 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.552731 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.552743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.562892 (+ 10149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.562898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.564719 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.564736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.571477 (+ 6741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.571481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.571585 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.571612 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.571678 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.571755 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7807,"thread_start_us":119,"threads_started":1}
I20260619 16:47:59.573130 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.544330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:40601 (local address 127.0.0.1:42022)
0619 16:47:59.551379 (+ 7049us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.551400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.551519 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.554447 (+ 2928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.554451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.554475 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.554629 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.554640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.558581 (+ 3941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.558588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.560836 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.560856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.572733 (+ 11877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.572737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.572845 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.572870 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.572943 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.573014 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6952,"thread_start_us":110,"threads_started":1}
I20260619 16:47:59.575656 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.539008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.3.233.130:33977)
0619 16:47:59.548084 (+ 9076us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.548092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.549172 (+ 1080us) server_negotiation.cc:407] Connection header received
0619 16:47:59.549275 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.549283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.549569 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.549754 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.555151 (+ 5397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.556704 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.568677 (+ 11973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.569794 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.569939 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.570133 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.575377 (+ 5244us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.575510 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8949,"thread_start_us":78,"threads_started":1}
I20260619 16:47:59.576238 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.544787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.3.233.131:40601)
0619 16:47:59.549096 (+ 4309us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.549111 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.551651 (+ 2540us) server_negotiation.cc:407] Connection header received
0619 16:47:59.551773 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.551781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.552042 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.552238 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.556924 (+ 4686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.558387 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.567049 (+ 8662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.568220 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.568384 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.568591 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.575968 (+ 7377us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.576096 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4171,"thread_start_us":108,"threads_started":1}
I20260619 16:47:59.577320 15388 tablet_service.cc:1467] Processing CreateTablet for tablet fe135c3a1cf54b35b5159be3fccc359a (DEFAULT_TABLE table=test-table [id=97ad100519474f3da9af4d80506154f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:59.578594 15388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe135c3a1cf54b35b5159be3fccc359a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:59.579301 15527 tablet_service.cc:1467] Processing CreateTablet for tablet fe135c3a1cf54b35b5159be3fccc359a (DEFAULT_TABLE table=test-table [id=97ad100519474f3da9af4d80506154f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:59.580300 15527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe135c3a1cf54b35b5159be3fccc359a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:59.587728 15747 tablet_bootstrap.cc:492] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44: Bootstrap starting.
I20260619 16:47:59.590281 15747 tablet_bootstrap.cc:654] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:59.591322 15747 log.cc:824] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:59.593596 15747 tablet_bootstrap.cc:492] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44: No bootstrap required, opened a new log
I20260619 16:47:59.593804 15747 ts_tablet_manager.cc:1397] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:47:59.594063 15656 tablet_service.cc:1467] Processing CreateTablet for tablet fe135c3a1cf54b35b5159be3fccc359a (DEFAULT_TABLE table=test-table [id=97ad100519474f3da9af4d80506154f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:47:59.597097 15656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe135c3a1cf54b35b5159be3fccc359a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:47:59.605047 15747 raft_consensus.cc:348] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.607721 15748 tablet_bootstrap.cc:492] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Bootstrap starting.
I20260619 16:47:59.611009 15747 raft_consensus.cc:374] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:59.611220 15747 raft_consensus.cc:729] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1da4e559854c11980aaae7e5740e44, State: Initialized, Role: FOLLOWER
I20260619 16:47:59.611596 15748 tablet_bootstrap.cc:654] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:59.611756 15750 tablet_bootstrap.cc:492] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Bootstrap starting.
I20260619 16:47:59.611949 15747 consensus_queue.cc:260] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.613044 15748 log.cc:824] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:59.613253 15747 ts_tablet_manager.cc:1428] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260619 16:47:59.613692 15457 heartbeater.cc:503] Master 127.3.233.190:34499 was elected leader, sending a full tablet report...
I20260619 16:47:59.614789 15750 tablet_bootstrap.cc:654] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Neither blocks nor log segments found. Creating new log.
I20260619 16:47:59.615453 15748 tablet_bootstrap.cc:492] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: No bootstrap required, opened a new log
I20260619 16:47:59.615667 15748 ts_tablet_manager.cc:1397] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:47:59.615836 15750 log.cc:824] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Log is configured to *not* fsync() on all Append() calls
I20260619 16:47:59.617965 15750 tablet_bootstrap.cc:492] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: No bootstrap required, opened a new log
I20260619 16:47:59.618156 15750 ts_tablet_manager.cc:1397] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:47:59.619797 15748 raft_consensus.cc:348] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.620100 15748 raft_consensus.cc:374] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:59.620162 15748 raft_consensus.cc:729] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63be1c223a2458bac2d87bb01715005, State: Initialized, Role: FOLLOWER
I20260619 16:47:59.620643 15748 consensus_queue.cc:260] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.621727 15748 ts_tablet_manager.cc:1428] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 16:47:59.622084 15593 heartbeater.cc:503] Master 127.3.233.190:34499 was elected leader, sending a full tablet report...
I20260619 16:47:59.622677 15750 raft_consensus.cc:348] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.623093 15750 raft_consensus.cc:374] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:47:59.623214 15750 raft_consensus.cc:729] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f96fa2b53449bebd9b5a265aad1b48, State: Initialized, Role: FOLLOWER
I20260619 16:47:59.623757 15750 consensus_queue.cc:260] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
Jun 19 16:47:59 dist-test-slave-cgwx krb5kdc[14767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:47:59.625061 15750 ts_tablet_manager.cc:1428] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260619 16:47:59.625452 15727 heartbeater.cc:503] Master 127.3.233.190:34499 was elected leader, sending a full tablet report...
W20260619 16:47:59.688169 15458 tablet.cc:2369] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:47:59.704167 15728 tablet.cc:2369] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:47:59.718237 15755 raft_consensus.cc:482] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:47:59.718514 15755 raft_consensus.cc:504] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.720161 15755 leader_election.cc:288] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63be1c223a2458bac2d87bb01715005 (127.3.233.130:33977), 6a1da4e559854c11980aaae7e5740e44 (127.3.233.129:42979)
I20260619 16:47:59.726485 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.720371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:33977 (local address 127.3.233.131:39065)
0619 16:47:59.720716 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.720734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.720839 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.721723 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.721726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.721746 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.721877 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.721884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.723068 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.723072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.724878 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.724890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.726002 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.726006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.726127 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.726153 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.726235 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.726326 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260619 16:47:59.727895 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.721066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:39065 (local address 127.3.233.130:33977)
0619 16:47:59.721358 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.721363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.721379 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:47:59.721452 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.721455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.721506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.721587 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.722018 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.722897 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.725044 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.725736 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.725875 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.725935 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.727648 (+ 1713us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.727754 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260619 16:47:59.729219 15547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe135c3a1cf54b35b5159be3fccc359a" candidate_uuid: "48f96fa2b53449bebd9b5a265aad1b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63be1c223a2458bac2d87bb01715005" is_pre_election: true
I20260619 16:47:59.729606 15547 raft_consensus.cc:2454] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f96fa2b53449bebd9b5a265aad1b48 in term 0.
I20260619 16:47:59.730397 15616 leader_election.cc:302] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f96fa2b53449bebd9b5a265aad1b48, b63be1c223a2458bac2d87bb01715005; no voters:
I20260619 16:47:59.730770 15755 raft_consensus.cc:2781] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:47:59.730854 15755 raft_consensus.cc:482] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:47:59.730899 15755 raft_consensus.cc:3037] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:59.732402 15755 raft_consensus.cc:504] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.732882 15755 leader_election.cc:288] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [CANDIDATE]: Term 1 election: Requested vote from peers b63be1c223a2458bac2d87bb01715005 (127.3.233.130:33977), 6a1da4e559854c11980aaae7e5740e44 (127.3.233.129:42979)
I20260619 16:47:59.733952 15547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe135c3a1cf54b35b5159be3fccc359a" candidate_uuid: "48f96fa2b53449bebd9b5a265aad1b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63be1c223a2458bac2d87bb01715005"
I20260619 16:47:59.734098 15547 raft_consensus.cc:3037] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:59.738126 15547 raft_consensus.cc:2454] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f96fa2b53449bebd9b5a265aad1b48 in term 1.
I20260619 16:47:59.740327 15616 leader_election.cc:302] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f96fa2b53449bebd9b5a265aad1b48, b63be1c223a2458bac2d87bb01715005; no voters:
I20260619 16:47:59.740639 15755 raft_consensus.cc:2781] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:47:59.740923 15755 raft_consensus.cc:686] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 1 LEADER]: Becoming Leader. State: Replica: 48f96fa2b53449bebd9b5a265aad1b48, State: Running, Role: LEADER
I20260619 16:47:59.741344 15755 consensus_queue.cc:237] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } }
I20260619 16:47:59.744385 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.720637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:57469 (local address 127.3.233.129:42979)
0619 16:47:59.721254 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.721259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.727198 (+ 5939us) server_negotiation.cc:407] Connection header received
0619 16:47:59.727266 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.727269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.727323 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.727400 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:47:59.736239 (+ 8839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.737167 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.739941 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.740595 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.740744 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.740810 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.744113 (+ 3303us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.744231 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501}
I20260619 16:47:59.745060 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.720501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42979 (local address 127.3.233.131:57469)
0619 16:47:59.726711 (+ 6210us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.726729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:47:59.726821 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:47:59.733767 (+ 6946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:47:59.733771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:47:59.733800 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:47:59.733911 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.733918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.738383 (+ 4465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.738388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.739775 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:47:59.739793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.743028 (+ 3235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.743033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:47:59.743148 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.743180 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:47:59.744790 (+ 1610us) client_negotiation.cc:240] Negotiation successful
0619 16:47:59.744889 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6135,"thread_start_us":145,"threads_started":1}
I20260619 16:47:59.745977 15411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe135c3a1cf54b35b5159be3fccc359a" candidate_uuid: "48f96fa2b53449bebd9b5a265aad1b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1da4e559854c11980aaae7e5740e44" is_pre_election: true
I20260619 16:47:59.746326 15411 raft_consensus.cc:2454] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f96fa2b53449bebd9b5a265aad1b48 in term 0.
I20260619 16:47:59.746685 15410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe135c3a1cf54b35b5159be3fccc359a" candidate_uuid: "48f96fa2b53449bebd9b5a265aad1b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1da4e559854c11980aaae7e5740e44"
I20260619 16:47:59.746805 15410 raft_consensus.cc:3037] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:47:59.747713 15241 catalog_manager.cc:5495] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48f96fa2b53449bebd9b5a265aad1b48 (127.3.233.131). New cstate: current_term: 1 leader_uuid: "48f96fa2b53449bebd9b5a265aad1b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48f96fa2b53449bebd9b5a265aad1b48" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 40601 } health_report { overall_health: HEALTHY } } }
I20260619 16:47:59.748237 15410 raft_consensus.cc:2454] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f96fa2b53449bebd9b5a265aad1b48 in term 1.
W20260619 16:47:59.771133 15594 tablet.cc:2369] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:47:59.793939 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.787988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.3.233.131:40601)
0619 16:47:59.788128 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:47:59.788134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:47:59.788152 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:47:59.788373 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:47:59.788377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:47:59.788457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:47:59.788597 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:47:59.789074 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.789959 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.791428 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:47:59.791758 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:47:59.791878 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:47:59.792352 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:47:59.793031 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:47:59.793177 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:47:59.793692 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 16:47:59.793767 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:47:59.801774 15410 raft_consensus.cc:1261] T fe135c3a1cf54b35b5159be3fccc359a P 6a1da4e559854c11980aaae7e5740e44 [term 1 FOLLOWER]: Refusing update from remote peer 48f96fa2b53449bebd9b5a265aad1b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:59.803093 15755 consensus_queue.cc:1035] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1da4e559854c11980aaae7e5740e44" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:59.803560 15547 raft_consensus.cc:1261] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 1 FOLLOWER]: Refusing update from remote peer 48f96fa2b53449bebd9b5a265aad1b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:47:59.807596 15755 consensus_queue.cc:1035] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63be1c223a2458bac2d87bb01715005" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 33977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:47:59.823004 15761 mvcc.cc:205] Tried to move back new op lower bound from 7298611936458743808 to 7298611936224690176. Current Snapshot: MvccSnapshot[applied={T|T < 7298611936458743808}]
I20260619 16:47:59.831389 15762 mvcc.cc:205] Tried to move back new op lower bound from 7298611936458743808 to 7298611936224690176. Current Snapshot: MvccSnapshot[applied={T|T < 7298611936458743808}]
I20260619 16:47:59.841624 15241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:59.841915 15241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:47:59.845132 15241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f6243b23e0d40208fc46ef16056fe9a: Sending DeleteTablet for 3 replicas of tablet fe135c3a1cf54b35b5159be3fccc359a
I20260619 16:47:59.846429 15388 tablet_service.cc:1514] Processing DeleteTablet for tablet fe135c3a1cf54b35b5159be3fccc359a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:59 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56770
I20260619 16:47:59.846608 15656 tablet_service.cc:1514] Processing DeleteTablet for tablet fe135c3a1cf54b35b5159be3fccc359a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:59 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:42022
I20260619 16:47:59.847514 15527 tablet_service.cc:1514] Processing DeleteTablet for tablet fe135c3a1cf54b35b5159be3fccc359a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:47:59 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:38546
I20260619 16:47:59.848438 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 15324
I20260619 16:47:59.855212 15774 tablet_replica.cc:330] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: stopping tablet replica
I20260619 16:47:59.855201 15773 tablet_replica.cc:330] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: stopping tablet replica
I20260619 16:47:59.855612 15773 raft_consensus.cc:2229] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:47:59.855612 15774 raft_consensus.cc:2229] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:47:59.856025 15774 raft_consensus.cc:2258] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:59.856160 15773 raft_consensus.cc:2258] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:47:59.858346 15773 ts_tablet_manager.cc:1905] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:47:59.863881 15774 ts_tablet_manager.cc:1905] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:47:59.864589 15230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:42979 (error 108)
I20260619 16:47:59.867226 15774 ts_tablet_manager.cc:1918] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:59.867357 15774 log.cc:1196] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/fe135c3a1cf54b35b5159be3fccc359a
I20260619 16:47:59.867712 15774 ts_tablet_manager.cc:1939] T fe135c3a1cf54b35b5159be3fccc359a P b63be1c223a2458bac2d87bb01715005: Deleting consensus metadata
I20260619 16:47:59.867887 15773 ts_tablet_manager.cc:1918] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:47:59.867997 15773 log.cc:1196] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/fe135c3a1cf54b35b5159be3fccc359a
I20260619 16:47:59.868343 15773 ts_tablet_manager.cc:1939] T fe135c3a1cf54b35b5159be3fccc359a P 48f96fa2b53449bebd9b5a265aad1b48: Deleting consensus metadata
I20260619 16:47:59.869289 15229 catalog_manager.cc:4841] TS b63be1c223a2458bac2d87bb01715005 (127.3.233.130:33977): tablet fe135c3a1cf54b35b5159be3fccc359a (table test-table [id=97ad100519474f3da9af4d80506154f2]) successfully deleted
I20260619 16:47:59.869462 15227 catalog_manager.cc:4841] TS 48f96fa2b53449bebd9b5a265aad1b48 (127.3.233.131:40601): tablet fe135c3a1cf54b35b5159be3fccc359a (table test-table [id=97ad100519474f3da9af4d80506154f2]) successfully deleted
I20260619 16:47:59.870064 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 15463
I20260619 16:47:59.872195 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:47:59.871785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42979 (local address 127.0.0.1:56774)
0619 16:47:59.872003 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:47:59.872086 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:42979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 16:47:59.872397 15230 catalog_manager.cc:4584] TS 6a1da4e559854c11980aaae7e5740e44 (127.3.233.129:42979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe135c3a1cf54b35b5159be3fccc359a: Network error: Client connection negotiation failed: client connection to 127.3.233.129:42979: connect: Connection refused (error 111)
I20260619 16:47:59.878413 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 15598
I20260619 16:47:59.887553 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 15211
I20260619 16:47:59.895100 4006 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:48:00.240923 4006 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:48:00.241113 4006 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:48:00.586870 4006 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:48:00.587 UTC [14797] LOG: received smart shutdown request
2026-06-19 16:48:00.590 UTC [14797] LOG: background worker "logical replication launcher" (PID 14806) exited with exit code 1
2026-06-19 16:48:00.591 UTC [14801] LOG: shutting down
2026-06-19 16:48:00.776 UTC [14797] LOG: database system is shut down
2026-06-19T16:48:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48259 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:00 dist-test-slave-cgwx krb5kdc[15784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:00 dist-test-slave-cgwx krb5kdc[15784](info): set up 2 sockets
Jun 19 16:48:00 dist-test-slave-cgwx krb5kdc[15784](info): commencing operation
krb5kdc: starting...
W20260619 16:48:01.975262 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:48:02 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/rangeradmin_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/rangeradmin_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/rangerlookup_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/rangerlookup_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
I20260619 16:48:02.158071 4006 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfHMdDR/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/postgres -l logfile start
2026-06-19 16:48:04.765 UTC [15814] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:48:04.765 UTC [15814] LOG: listening on IPv4 address "127.3.233.148", port 54627
2026-06-19 16:48:04.768 UTC [15814] LOG: listening on Unix socket "/tmp/.s.PGSQL.54627"
2026-06-19 16:48:04.803 UTC [15817] LOG: database system was shut down at 2026-06-19 16:48:03 UTC
2026-06-19 16:48:04.807 UTC [15814] LOG: database system is ready to accept connections
I20260619 16:48:05.766891 4006 mini_postgres.cc:96] Postgres bound to 54627
2026-06-19 16:48:05.773 UTC [15825] FATAL: database "slave" does not exist
127.3.233.148:54627 - accepting connections
I20260619 16:48:05.773779 4006 mini_ranger.cc:161] Starting Ranger...
I20260619 16:48:05.788739 4006 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:48:06.285964 4006 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:48:06.286051 4006 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-admin
2026-06-19 16:48:06,556 [I] DB FLAVOR :POSTGRES
2026-06-19 16:48:06,556 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:48:06,556 [I] Checking connection..
2026-06-19 16:48:06,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:48:06,805 [I] Checking connection passed.
2026-06-19 16:48:06,805 [I] --------- Verifying version history table ---------
2026-06-19 16:48:06,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:48:07,066 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:48:07,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:48:07,314 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:48:07,314 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:48:07,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:48:07.545 UTC [15889] WARNING: there is no transaction in progress
2026-06-19 16:48:07,552 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:48:07,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:48:07,810 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:48:07,811 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:48:07,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:48:08,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:48:08,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cgwx.c.gcp-upstream.internal','N') ;"
2026-06-19 16:48:08,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:48:08,775 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:48:08,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:48:09,037 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:48:09,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:48:09,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:48:09,510 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:48:09,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:48:10.490 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.506 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.522 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.538 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.554 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.571 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.630 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.636 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.651 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.662 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.672 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.685 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.695 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.702 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.710 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:10.717 UTC [15997] WARNING: there is no transaction in progress
2026-06-19 16:48:11,406 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:48:11,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:48:11,655 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:48:11,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:48:11,893 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:48:11,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:48:12,137 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:48:12,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:48:12,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:48:12,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cgwx.c.gcp-upstream.internal';"
2026-06-19 16:48:12,852 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:48:12,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:48:13,110 [I] DB_PATCHES have already been applied
I20260619 16:48:13.116145 4006 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfHMdDR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:48:13.116217 4006 mini_ranger.cc:193] Using host: 127.3.233.148
I20260619 16:48:13.119721 4006 mini_ranger.cc:227] Ranger admin URL: http://127.3.233.148:33719
Jun 19, 2026 4:48:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:48:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:48:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:48:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:48:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:48:14.237928 4006 mini_ranger.cc:160] Time spent starting Ranger: real 8.464s user 0.007s sys 0.003s
Jun 19, 2026 4:48:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:48:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:48:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:48:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:48:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:48:36.410213 4006 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.233.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.233.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/rangerkms_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.233.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/rangerkms_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.233.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.233.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.233.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/HTTP_127.3.233.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:48:36 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887716, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:48:36.492707 4006 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:48:36.502251 4006 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:48:37.414017 4006 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:48:37.414110 4006 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms
I20260619 16:48:37.414134 4006 mini_ranger_kms.cc:227] Using postgres at 127.3.233.148:54627
2026-06-19 16:48:37,583 [I] DB FLAVOR :POSTGRES
2026-06-19 16:48:37,583 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:48:37,583 [I] Checking connection
2026-06-19 16:48:37,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:48:37,832 [I] connection success
2026-06-19 16:48:37,833 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:48:37,833 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:48:37,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:48:38,066 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:48:38,066 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:48:38,066 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:48:38,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.233.148:54627/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:48:38,324 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:48:38.804371 4006 mini_ranger_kms.cc:310] Created kms service
I20260619 16:48:38.987373 4006 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:48:39.030417 4006 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:48:39 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887716, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:39.425944 4006 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:48:39.426198 4006 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfHMdDR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfHMdDR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:48:39.426229 4006 mini_ranger_kms.cc:240] Using host: 127.3.233.148
I20260619 16:48:39.429821 4006 mini_ranger_kms.cc:276] Ranger KMS PID: 16177
I20260619 16:48:39.429901 4006 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.233.148:43073
Jun 19, 2026 4:48:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:48:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:48:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:48:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:48:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:48:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 16:48:41 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.233.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:41.758358 4006 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.266s user 0.000s sys 0.154s
I20260619 16:48:41.758486 4006 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:48:41.758555 4006 mini_ranger_kms.cc:401] 127.3.233.148:43073/kms/v1/keys
Jun 19 16:48:42 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.233.148@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
Jun 19 16:48:42 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887721, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.233.148@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
Jun 19 16:48:44 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:48:44.882201 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34343
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34343
--ranger_config_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:44.999701 16228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:44.999995 16228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:45.000036 16228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:45.004199 16228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:45.004266 16228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:45.004287 16228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:45.004302 16228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:45.004347 16228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:45.010004 16228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.233.148:43073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34343
--ranger_config_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:45.011283 16228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:45.012394 16228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:45.021790 16233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:45.021767 16234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:45.021767 16236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:45.022161 16228 server_base.cc:1034] running on GCE node
I20260619 16:48:45.022742 16228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:45.024016 16228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:45.025219 16228 hybrid_clock.cc:648] HybridClock initialized: now 1781887725025196 us; error 45 us; skew 500 ppm
Jun 19 16:48:45 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:45.028373 16228 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:48:45.029476 16228 webserver.cc:460] Webserver started at http://127.3.233.190:42383/ using document root <none> and password file <none>
I20260619 16:48:45.030135 16228 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:45.030212 16228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:45.030448 16228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:48:45 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887725, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:45.092568 16228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "45b28ac44c3b4d8493f31595b9a81ee8"
format_stamp: "Formatted at 2026-06-19 16:48:45 on dist-test-slave-cgwx"
server_key: "7f2f6edd9f938dca8bc63f159d170bbc"
server_key_iv: "1442737645cfd88deab066a92d3655df"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:45.093219 16228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "45b28ac44c3b4d8493f31595b9a81ee8"
format_stamp: "Formatted at 2026-06-19 16:48:45 on dist-test-slave-cgwx"
server_key: "7f2f6edd9f938dca8bc63f159d170bbc"
server_key_iv: "1442737645cfd88deab066a92d3655df"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:45.097437 16228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:48:45.113404 16243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:45.115248 16228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:45.115410 16228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "45b28ac44c3b4d8493f31595b9a81ee8"
format_stamp: "Formatted at 2026-06-19 16:48:45 on dist-test-slave-cgwx"
server_key: "7f2f6edd9f938dca8bc63f159d170bbc"
server_key_iv: "1442737645cfd88deab066a92d3655df"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:45.115558 16228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:45.131994 16228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:45.135116 16228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:45.135335 16228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:45.143715 16228 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34343
I20260619 16:48:45.143715 16295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34343 every 8 connection(s)
I20260619 16:48:45.144747 16228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:45.147647 16296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:45.150149 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 16228
I20260619 16:48:45.150287 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:48:45.150604 4006 external_mini_cluster.cc:1428] Setting key 550544f7b5b9a7e0a1ec153fb73d2196
I20260619 16:48:45.153926 16296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Bootstrap starting.
I20260619 16:48:45.156772 16296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:45.157733 16296 log.cc:824] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:45.160046 16296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: No bootstrap required, opened a new log
Jun 19 16:48:45 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:45.164608 16296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:45.164904 16296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:45.165009 16296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b28ac44c3b4d8493f31595b9a81ee8, State: Initialized, Role: FOLLOWER
I20260619 16:48:45.165551 16296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:45.165740 16296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:45.165836 16296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:45.165941 16296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:45.167217 16296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:45.167600 16296 leader_election.cc:302] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45b28ac44c3b4d8493f31595b9a81ee8; no voters:
I20260619 16:48:45.167989 16296 leader_election.cc:288] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:45.168062 16301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:45.168164 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:45.152215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.3.233.190:34343)
0619 16:48:45.152743 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:45.152764 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:45.152808 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:48:45.153598 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:45.153620 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:45.154040 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:45.154420 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:45.155359 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:45.156771 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:45.158010 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:45.158776 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:45.159001 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:45.162308 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:45.162315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:45.162336 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:45.162369 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:45.164482 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:45.165060 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:45.165067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:45.165074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:45.165217 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:45.165596 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:45.165600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:45.165602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:45.166167 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:45.166404 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:45.166695 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 16:48:45.167034 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":158,"threads_started":1}
I20260619 16:48:45.168320 16301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 45b28ac44c3b4d8493f31595b9a81ee8, State: Running, Role: LEADER
I20260619 16:48:45.168820 16301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:45.169337 16296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:45.170168 16301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } } }
I20260619 16:48:45.170279 16301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:45.170588 16302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45b28ac44c3b4d8493f31595b9a81ee8. Latest consensus state: current_term: 1 leader_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } } }
I20260619 16:48:45.170764 16302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:45.171602 16296 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:48:45.175431 16310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:45.179621 16310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:45.187996 16310 catalog_manager.cc:1347] Generated new cluster ID: 96d7aadf4b764a78888acc687100ecca
I20260619 16:48:45.188086 16310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:45.225944 16310 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:45.227411 16310 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:45.234743 16310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Generated new TSK 0
I20260619 16:48:45.235702 16310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:48:45.982843 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34343
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:46.125921 16341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:46.126181 16341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:46.126224 16341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:46.130462 16341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:46.130542 16341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:46.130625 16341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:48:46.138612 16341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.233.148:43073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:46.140154 16341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:46.141562 16341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:46.155370 16349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.155375 16348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:46.156337 16341 server_base.cc:1034] running on GCE node
W20260619 16:48:46.156741 16351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:46.157508 16341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:46.158375 16341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:46.159622 16341 hybrid_clock.cc:648] HybridClock initialized: now 1781887726159603 us; error 38 us; skew 500 ppm
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:46.162876 16341 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:48:46.164007 16341 webserver.cc:460] Webserver started at http://127.3.233.129:35143/ using document root <none> and password file <none>
I20260619 16:48:46.164615 16341 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:46.164718 16341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:46.165024 16341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:46.192916 16341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "1ed0ebd2425c423ebd82e4c31f8f2848"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6d6088e79493211ba7905f03a4d8acb3"
server_key_iv: "4455010a6e778b0d3895555e2609e49e"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.193753 16341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "1ed0ebd2425c423ebd82e4c31f8f2848"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6d6088e79493211ba7905f03a4d8acb3"
server_key_iv: "4455010a6e778b0d3895555e2609e49e"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.199244 16341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:46.218498 16358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.220414 16341 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 16:48:46.220640 16341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "1ed0ebd2425c423ebd82e4c31f8f2848"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6d6088e79493211ba7905f03a4d8acb3"
server_key_iv: "4455010a6e778b0d3895555e2609e49e"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.220868 16341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:46.242995 16341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:46.246803 16341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:46.247098 16341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:46.247943 16341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:46.249037 16341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:46.249141 16341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.249265 16341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:46.249342 16341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.260357 16341 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:43975
I20260619 16:48:46.260473 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:43975 every 8 connection(s)
I20260619 16:48:46.261423 16341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:48:46.265327 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 16341
I20260619 16:48:46.265436 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:48:46.265728 4006 external_mini_cluster.cc:1428] Setting key 474aa2cdbeb90b318dba75298ef28699
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:46.285746 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.265957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:60231 (local address 127.3.233.190:34343)
0619 16:48:46.269488 (+ 3531us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.269492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.270191 (+ 699us) server_negotiation.cc:407] Connection header received
0619 16:48:46.272857 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.272860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.272918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.272999 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.274580 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.275526 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.277215 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.277619 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.277699 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.280984 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.280988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.280991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.281020 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.283049 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.283762 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.283766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.283768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.283860 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.284290 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.284293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.284295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.284545 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.284619 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.285401 (+ 782us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.285550 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3438,"thread_start_us":79,"threads_started":1}
I20260619 16:48:46.288003 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.264758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34343 (local address 127.3.233.129:60231)
0619 16:48:46.269791 (+ 5033us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.269858 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.271105 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.273316 (+ 2211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.273327 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.273713 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.274404 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.274425 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.275672 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.275676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.277070 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.277082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.277916 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.277919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.278117 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.278854 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.278877 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.280766 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.283213 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.283219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.283240 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.283632 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.283976 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.283979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.283981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.284176 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.284703 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.284710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.285089 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.286346 (+ 1257us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.286733 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4790,"thread_start_us":139,"threads_started":1}
I20260619 16:48:46.289736 16474 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34343
I20260619 16:48:46.289981 16474 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:46.290567 16474 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:48:46.292418 16260 ts_manager.cc:194] Registered new tserver with Master: 1ed0ebd2425c423ebd82e4c31f8f2848 (127.3.233.129:43975)
I20260619 16:48:46.293959 16260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:60231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:48:46.316780 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34343
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:48:46.455564 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:46.455883 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:46.456012 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:46.460685 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:46.460789 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:46.460899 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:48:46.466995 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.233.148:43073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:46.468230 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:46.469307 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:46.479538 16487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.479671 16489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.479538 16486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:46.480381 16480 server_base.cc:1034] running on GCE node
I20260619 16:48:46.480844 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:46.481520 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:46.482686 16480 hybrid_clock.cc:648] HybridClock initialized: now 1781887726482612 us; error 89 us; skew 500 ppm
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:46.486032 16480 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:48:46.487140 16480 webserver.cc:460] Webserver started at http://127.3.233.130:41517/ using document root <none> and password file <none>
I20260619 16:48:46.487923 16480 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:46.488091 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:46.488387 16480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:46.518882 16480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "b0351a65e48741d4a64d17a7e6278ed9"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "c35dc330c618eeab6b73680a959842a1"
server_key_iv: "491e88653d91b0fdce6e6999a4520e55"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.519838 16480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "b0351a65e48741d4a64d17a7e6278ed9"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "c35dc330c618eeab6b73680a959842a1"
server_key_iv: "491e88653d91b0fdce6e6999a4520e55"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.527340 16480 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 16:48:46.551403 16496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.553359 16480 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260619 16:48:46.553524 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "b0351a65e48741d4a64d17a7e6278ed9"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "c35dc330c618eeab6b73680a959842a1"
server_key_iv: "491e88653d91b0fdce6e6999a4520e55"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.553671 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:46.574612 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:46.578485 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:46.578744 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:46.579523 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:46.580651 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:46.580755 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.580850 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:46.580915 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.603030 16480 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:37167
I20260619 16:48:46.604379 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:46.606848 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 16480
I20260619 16:48:46.606986 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:48:46.607334 4006 external_mini_cluster.cc:1428] Setting key e977e91aec32c48141594220bfb2688b
I20260619 16:48:46.605378 16609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:37167 every 8 connection(s)
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:48:46.634583 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.609226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:58061 (local address 127.3.233.190:34343)
0619 16:48:46.609357 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:46.609362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:46.610312 (+ 950us) server_negotiation.cc:407] Connection header received
0619 16:48:46.611784 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:46.611788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:46.611844 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:46.611898 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:46.613687 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.614539 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.616456 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.616816 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.616915 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.623847 (+ 6932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:46.623852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:46.623855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:46.623892 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:46.625786 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.626695 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.626700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.626702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.626798 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:46.632820 (+ 6022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:46.632827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:46.632829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:46.633208 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:46.633352 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:46.634231 (+ 879us) server_negotiation.cc:299] Negotiation successful
0619 16:48:46.634402 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:48:46.636488 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:46.608015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34343 (local address 127.3.233.130:58061)
0619 16:48:46.610141 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:46.610206 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:46.611511 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:46.612366 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:46.612376 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:46.612764 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:46.613494 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.613515 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.614738 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.614741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.616299 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:46.616311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:46.617158 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:46.617162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:46.617401 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:46.621457 (+ 4056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:46.621484 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:46.623549 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:46.626103 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.626111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.626135 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.626559 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.626904 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:46.626906 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:46.626908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:46.627245 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:46.633454 (+ 6209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:46.633464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:46.633883 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 16:48:46.634855 (+ 972us) client_negotiation.cc:240] Negotiation successful
0619 16:48:46.635314 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1860,"thread_start_us":158,"threads_started":1}
I20260619 16:48:46.638265 16610 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34343
I20260619 16:48:46.638550 16610 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:46.639209 16610 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
I20260619 16:48:46.640585 16260 ts_manager.cc:194] Registered new tserver with Master: b0351a65e48741d4a64d17a7e6278ed9 (127.3.233.130:37167)
I20260619 16:48:46.641343 16260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:58061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:48:46.658382 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34343
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:46.846659 16615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:46.847002 16615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:46.847111 16615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:46.852058 16615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:46.852154 16615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:46.852273 16615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:46.859848 16615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.233.148:43073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:46.861490 16615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:46.862922 16615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:46.884735 16621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:46.884742 16620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:46.885290 16615 server_base.cc:1034] running on GCE node
W20260619 16:48:46.890054 16623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:46.890682 16615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:46.891619 16615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:46.892823 16615 hybrid_clock.cc:648] HybridClock initialized: now 1781887726892797 us; error 40 us; skew 500 ppm
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:46.896533 16615 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:48:46.897665 16615 webserver.cc:460] Webserver started at http://127.3.233.131:42695/ using document root <none> and password file <none>
I20260619 16:48:46.898393 16615 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:46.898484 16615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:46.898731 16615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:48:46 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:46.914894 16615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "7efe8c4f7b3e491a8273b03af60c2401"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6c9ffda9ddfd1f6c286bd0a69f91b1d8"
server_key_iv: "73a180ceee995e4268eb31348b271a83"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.916059 16615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "7efe8c4f7b3e491a8273b03af60c2401"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6c9ffda9ddfd1f6c286bd0a69f91b1d8"
server_key_iv: "73a180ceee995e4268eb31348b271a83"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.941303 16615 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.004s sys 0.000s
I20260619 16:48:46.965715 16630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:46.967283 16615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:46.967451 16615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "7efe8c4f7b3e491a8273b03af60c2401"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6c9ffda9ddfd1f6c286bd0a69f91b1d8"
server_key_iv: "73a180ceee995e4268eb31348b271a83"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:46.967592 16615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:46.999388 16615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:47.002542 16615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:47.002753 16615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:47.003468 16615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:47.004412 16615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:47.004475 16615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.004526 16615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:47.004594 16615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 16:48:47.015641 16615 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:43695
I20260619 16:48:47.015664 16743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:43695 every 8 connection(s)
I20260619 16:48:47.016985 16615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:47.017303 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 16615
I20260619 16:48:47.017433 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:48:47.017709 4006 external_mini_cluster.cc:1428] Setting key 46b5d783f7d735460241fa8cb5bb9bf2
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887726, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:47.036264 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:47.022017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:52089 (local address 127.3.233.190:34343)
0619 16:48:47.022158 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.022163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.022179 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:48:47.023138 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.023141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.023194 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.023245 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.024982 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.025881 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.027674 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.028036 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.028154 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:47.031794 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:47.031799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:47.031801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:47.031830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:47.033650 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:47.034320 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:47.034324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:47.034327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:47.034427 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:47.034828 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:47.034831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:47.034833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:47.035109 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:47.035192 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:47.035958 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 16:48:47.036100 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 16:48:47.038007 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:47.020845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34343 (local address 127.3.233.131:52089)
0619 16:48:47.021498 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:47.021563 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:47.022850 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:47.023701 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:47.023710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:47.024098 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:47.024785 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:47.024803 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.026041 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.026045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:47.027531 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:47.027543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.028219 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.028223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:47.028456 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:47.029184 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:47.029206 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:47.031588 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:47.033792 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:47.033800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:47.033819 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:47.034196 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:47.034535 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:47.034538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:47.034540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:47.034725 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:47.035260 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:47.035268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:47.035633 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 16:48:47.036517 (+ 884us) client_negotiation.cc:240] Negotiation successful
0619 16:48:47.036881 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":189,"threads_started":1}
I20260619 16:48:47.039489 16744 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34343
I20260619 16:48:47.039781 16744 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:47.040372 16744 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
I20260619 16:48:47.041620 16260 ts_manager.cc:194] Registered new tserver with Master: 7efe8c4f7b3e491a8273b03af60c2401 (127.3.233.131:43695)
I20260619 16:48:47.042414 16260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:52089
I20260619 16:48:47.043988 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:48:47.044741 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 16341
I20260619 16:48:47.053218 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 16480
I20260619 16:48:47.060439 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 16615
I20260619 16:48:47.067842 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 16228
I20260619 16:48:47.076148 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34343
--webserver_interface=127.3.233.190
--webserver_port=42383
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34343
--ranger_config_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:47.188779 16747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:47.189056 16747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:47.189097 16747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:47.193364 16747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:47.193439 16747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:47.193457 16747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:47.193475 16747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:47.193490 16747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:47.199317 16747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34343
--ranger_config_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=42383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:47.200534 16747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:47.201642 16747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:47.210274 16753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:47.210318 16752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:47.210279 16755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:47.210690 16747 server_base.cc:1034] running on GCE node
I20260619 16:48:47.211330 16747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:47.212464 16747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:47.213665 16747 hybrid_clock.cc:648] HybridClock initialized: now 1781887727213645 us; error 35 us; skew 500 ppm
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:47.216759 16747 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:48:47.217697 16747 webserver.cc:460] Webserver started at http://127.3.233.190:42383/ using document root <none> and password file <none>
I20260619 16:48:47.218266 16747 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:47.218335 16747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:47.300721 16747 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:48:47.303535 16763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.305178 16747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:47.305321 16747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "45b28ac44c3b4d8493f31595b9a81ee8"
format_stamp: "Formatted at 2026-06-19 16:48:45 on dist-test-slave-cgwx"
server_key: "7f2f6edd9f938dca8bc63f159d170bbc"
server_key_iv: "1442737645cfd88deab066a92d3655df"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:47.306106 16747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:47.328814 16747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:47.332036 16747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:47.332198 16747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:47.340195 16747 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34343
I20260619 16:48:47.340209 16815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34343 every 8 connection(s)
I20260619 16:48:47.341361 16747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:47.344524 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 16747
I20260619 16:48:47.345041 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:43975
--local_ip_for_outbound_sockets=127.3.233.129
--tserver_master_addrs=127.3.233.190:34343
--webserver_port=35143
--webserver_interface=127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:47.346208 16816 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:48:47.348040 16816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Bootstrap starting.
I20260619 16:48:47.364779 16816 log.cc:824] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:47.371762 16816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:48:47.372265 16816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Bootstrap complete.
I20260619 16:48:47.376542 16816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:47.376952 16816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b28ac44c3b4d8493f31595b9a81ee8, State: Initialized, Role: FOLLOWER
I20260619 16:48:47.377455 16816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:47.377569 16816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:47.377660 16816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:47.377758 16816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:48:47.379894 16816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:47.380232 16816 leader_election.cc:302] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45b28ac44c3b4d8493f31595b9a81ee8; no voters:
I20260619 16:48:47.380582 16816 leader_election.cc:288] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:48:47.380667 16820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:48:47.380932 16820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [term 2 LEADER]: Becoming Leader. State: Replica: 45b28ac44c3b4d8493f31595b9a81ee8, State: Running, Role: LEADER
I20260619 16:48:47.381304 16820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } }
I20260619 16:48:47.381841 16816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:47.382771 16822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45b28ac44c3b4d8493f31595b9a81ee8. Latest consensus state: current_term: 2 leader_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } } }
I20260619 16:48:47.382817 16821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b28ac44c3b4d8493f31595b9a81ee8" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34343 } } }
I20260619 16:48:47.382910 16822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:47.382966 16821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:47.384328 16816 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:48:47.384975 16829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:47.391884 16829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:47.392624 16829 catalog_manager.cc:1259] Loaded cluster ID: 96d7aadf4b764a78888acc687100ecca
I20260619 16:48:47.392765 16829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:47.395771 16829 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:47.396508 16829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Loaded TSK: 0
I20260619 16:48:47.397789 16829 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:48:47.492835 16818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:47.493332 16818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:47.493515 16818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:47.500365 16818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:47.500464 16818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:47.500560 16818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:48:47.509500 16818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:43975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=35143
--tserver_master_addrs=127.3.233.190:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:47.511086 16818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:47.512552 16818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:47.526279 16856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:47.526371 16854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:47.526388 16853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:47.527786 16818 server_base.cc:1034] running on GCE node
I20260619 16:48:47.528405 16818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:47.529287 16818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:47.535722 16818 hybrid_clock.cc:648] HybridClock initialized: now 1781887727535677 us; error 53 us; skew 500 ppm
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:47.539657 16818 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:48:47.540845 16818 webserver.cc:460] Webserver started at http://127.3.233.129:35143/ using document root <none> and password file <none>
I20260619 16:48:47.541565 16818 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:47.541699 16818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:47.572549 16818 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:48:47.575716 16864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.577481 16818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:48:47.577641 16818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "1ed0ebd2425c423ebd82e4c31f8f2848"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6d6088e79493211ba7905f03a4d8acb3"
server_key_iv: "4455010a6e778b0d3895555e2609e49e"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:47.578413 16818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:47.621817 16818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:47.625756 16818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:47.625978 16818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:47.626701 16818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:47.627787 16818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:47.627864 16818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.627948 16818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:47.627985 16818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.654488 16818 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:43975
I20260619 16:48:47.654516 16977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:43975 every 8 connection(s)
I20260619 16:48:47.655915 16818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:48:47.657052 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 16818
I20260619 16:48:47.657603 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:37167
--local_ip_for_outbound_sockets=127.3.233.130
--tserver_master_addrs=127.3.233.190:34343
--webserver_port=41517
--webserver_interface=127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:47.687021 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:47.663337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:52189 (local address 127.3.233.190:34343)
0619 16:48:47.664184 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.664212 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.664854 (+ 642us) server_negotiation.cc:407] Connection header received
0619 16:48:47.666738 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.666756 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.667321 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.667792 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.669869 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.671677 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.673567 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.674391 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.674564 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:47.677980 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:47.677988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:47.678008 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:47.678046 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:47.680353 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:47.681143 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:47.681151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:47.681158 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:47.681337 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:47.681942 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:47.681946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:47.681949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:47.683458 (+ 1509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:47.683593 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:47.685618 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0619 16:48:47.685977 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":152,"threads_started":1}
I20260619 16:48:47.688704 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:47.663548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34343 (local address 127.3.233.129:52189)
0619 16:48:47.664575 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:47.664650 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:47.666096 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:47.668424 (+ 2328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:47.668436 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:47.668851 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:47.669620 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:47.669648 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.671871 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.671875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:47.673372 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:47.673387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.674815 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.674820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:47.675078 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:47.675888 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:47.675911 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:47.677606 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:47.680501 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:47.680509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:47.680530 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:47.680985 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:47.681486 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:47.681490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:47.681492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:47.681802 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:47.683665 (+ 1863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:47.683672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:47.683999 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 16:48:47.687452 (+ 3453us) client_negotiation.cc:240] Negotiation successful
0619 16:48:47.687822 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":171,"threads_started":1}
I20260619 16:48:47.692085 16978 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34343
I20260619 16:48:47.692320 16978 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:47.692914 16978 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
W20260619 16:48:47.695184 16978 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:34343 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:48:47.695302 16978 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:47.695478 16978 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
I20260619 16:48:47.696398 16978 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:47.696553 16978 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
W20260619 16:48:47.697359 16978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:48:47.796388 16980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:47.796726 16980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:47.796814 16980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:47.801276 16980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:47.801370 16980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:47.801483 16980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:48:47.807672 16980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:37167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=41517
--tserver_master_addrs=127.3.233.190:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:47.808970 16980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:47.810021 16980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:47.822909 16987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:47.822909 16988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:47.822909 16990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:47.823815 16980 server_base.cc:1034] running on GCE node
I20260619 16:48:47.824375 16980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:47.825340 16980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:47.826576 16980 hybrid_clock.cc:648] HybridClock initialized: now 1781887727826539 us; error 47 us; skew 500 ppm
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:47.830668 16980 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:48:47.832085 16980 webserver.cc:460] Webserver started at http://127.3.233.130:41517/ using document root <none> and password file <none>
I20260619 16:48:47.832908 16980 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:47.833041 16980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:47.864151 16980 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 16:48:47.867693 16998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.869374 16980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:48:47.869539 16980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "b0351a65e48741d4a64d17a7e6278ed9"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "c35dc330c618eeab6b73680a959842a1"
server_key_iv: "491e88653d91b0fdce6e6999a4520e55"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:47.870491 16980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:47.913111 16980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:47.917268 16980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:47.917680 16980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:47.918516 16980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:47.919749 16980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:47.919871 16980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.919982 16980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:47.920058 16980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:47.932143 16980 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:37167
I20260619 16:48:47.932204 17112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:37167 every 8 connection(s)
I20260619 16:48:47.933810 16980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:47.938119 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 16980
I20260619 16:48:47.938652 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.233.148:43073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:43695
--local_ip_for_outbound_sockets=127.3.233.131
--tserver_master_addrs=127.3.233.190:34343
--webserver_port=42695
--webserver_interface=127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:48:47 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:47.951828 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:47.936225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:41417 (local address 127.3.233.190:34343)
0619 16:48:47.936365 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:47.936369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:47.938996 (+ 2627us) server_negotiation.cc:407] Connection header received
0619 16:48:47.939795 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:47.939799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:47.939869 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:47.939979 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:47.941575 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.942449 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.944125 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.944480 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.944638 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:47.947164 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:47.947168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:47.947171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:47.947204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:47.948876 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:47.949570 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:47.949573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:47.949575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:47.949667 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:47.950099 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:47.950103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:47.950105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:47.950338 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:47.950409 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:47.951491 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0619 16:48:47.951642 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:48:47.953951 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:47.936761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34343 (local address 127.3.233.130:41417)
0619 16:48:47.938083 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:47.938149 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:47.939544 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:47.940209 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:47.940221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:47.940615 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:47.941379 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:47.941402 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.942620 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.942623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:47.943974 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:47.943985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:47.944673 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:47.944677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:47.944874 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:47.945523 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:47.945540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:47.946906 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:47.949042 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:47.949049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:47.949068 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:47.949437 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:47.949778 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:47.949781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:47.949782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:47.949983 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:47.950780 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:47.950787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:47.951178 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 16:48:47.952320 (+ 1142us) client_negotiation.cc:240] Negotiation successful
0619 16:48:47.952712 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":156,"threads_started":1}
I20260619 16:48:47.955544 17113 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34343
I20260619 16:48:47.955830 17113 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:47.956511 17113 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
W20260619 16:48:47.958248 17113 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:34343 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:48:47.958393 17113 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:47.958585 17113 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
I20260619 16:48:47.959532 17113 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:47.959717 17113 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
W20260619 16:48:47.960498 17113 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:48:48.104761 17116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:48.105190 17116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:48.105258 17116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:48.111865 17116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:48.112010 17116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:48.112114 17116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:48:48.120445 17116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.233.148:43073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:43695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=42695
--tserver_master_addrs=127.3.233.190:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:48.121903 17116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:48.123298 17116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:48.135407 17124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:48.135664 17122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:48.135826 17121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:48.136464 17116 server_base.cc:1034] running on GCE node
I20260619 16:48:48.137073 17116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:48.137877 17116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:48.139043 17116 hybrid_clock.cc:648] HybridClock initialized: now 1781887728139008 us; error 57 us; skew 500 ppm
Jun 19 16:48:48 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887728, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:48.142354 17116 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:48:48.143479 17116 webserver.cc:460] Webserver started at http://127.3.233.131:42695/ using document root <none> and password file <none>
I20260619 16:48:48.144172 17116 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:48.144295 17116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:48:48 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887728, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:48.166980 17116 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:48:48.170737 17132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:48.172691 17116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:48:48.172850 17116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "7efe8c4f7b3e491a8273b03af60c2401"
format_stamp: "Formatted at 2026-06-19 16:48:46 on dist-test-slave-cgwx"
server_key: "6c9ffda9ddfd1f6c286bd0a69f91b1d8"
server_key_iv: "73a180ceee995e4268eb31348b271a83"
server_key_version: "kuduclusterkey@0"
I20260619 16:48:48.173563 17116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:48.199740 17116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:48.203289 17116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:48.203536 17116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:48.204262 17116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:48.205235 17116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:48.205286 17116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:48.205325 17116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:48.205377 17116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:48.217684 17116 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:43695
I20260619 16:48:48.217733 17245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:43695 every 8 connection(s)
I20260619 16:48:48.219825 17116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:48.229163 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17116
Jun 19 16:48:48 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887728, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:48.235395 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.222012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:52185 (local address 127.3.233.190:34343)
0619 16:48:48.223792 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.223799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.223821 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:48:48.224525 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.224528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.224586 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.224679 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:48.225871 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.226412 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.227699 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.227977 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.228101 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.230915 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:48.230951 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:48.230958 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:48.230993 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:48.232947 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:48.233483 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:48.233487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:48.233488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:48.233580 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:48.233954 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:48.233958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:48.233960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:48.234179 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:48.234283 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.235061 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.235205 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689}
I20260619 16:48:48.236398 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.222653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34343 (local address 127.3.233.131:52185)
0619 16:48:48.223236 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:48.223293 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:48.224283 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:48.224878 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:48.224886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:48.225191 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:48.225711 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:48.225726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.226549 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.226552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:48.227560 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:48.227570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.228150 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.228154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:48.228343 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.229032 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:48.229049 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:48.230721 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:48.233043 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:48.233049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:48.233067 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:48.233357 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:48.233683 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:48.233686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:48.233688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:48.233850 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:48.234316 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:48.234324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:48.234694 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 16:48:48.234960 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 16:48:48.235332 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":175,"threads_started":1}
I20260619 16:48:48.237931 17246 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34343
I20260619 16:48:48.238184 17246 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:48.238777 17246 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
I20260619 16:48:48.240293 16774 ts_manager.cc:194] Registered new tserver with Master: 7efe8c4f7b3e491a8273b03af60c2401 (127.3.233.131:43695)
I20260619 16:48:48.241603 16774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:52185
I20260619 16:48:48.247884 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.229943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.3.233.190:34343)
0619 16:48:48.230257 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.230263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.231023 (+ 760us) server_negotiation.cc:407] Connection header received
0619 16:48:48.231163 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.231166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.231220 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.231342 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:48.239081 (+ 7739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.239844 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.241262 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.241589 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.241707 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.243883 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:48.243887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:48.243890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:48.243916 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:48.245745 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:48.246324 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:48.246327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:48.246328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:48.246427 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:48.246755 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:48.246758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:48.246760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:48.247098 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:48.247270 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.247495 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.247674 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
Jun 19 16:48:48 dist-test-slave-cgwx krb5kdc[15784](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887728, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:48.697589 16978 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:48.697906 16978 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
I20260619 16:48:48.698580 16774 ts_manager.cc:194] Registered new tserver with Master: 1ed0ebd2425c423ebd82e4c31f8f2848 (127.3.233.129:43975)
I20260619 16:48:48.699293 16774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:52189
I20260619 16:48:48.960716 17113 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:48.961032 17113 heartbeater.cc:511] Master 127.3.233.190:34343 requested a full tablet report, sending...
I20260619 16:48:48.962097 16774 ts_manager.cc:194] Registered new tserver with Master: b0351a65e48741d4a64d17a7e6278ed9 (127.3.233.130:37167)
I20260619 16:48:48.962826 16774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:41417
I20260619 16:48:48.967132 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:48:48 dist-test-slave-cgwx krb5kdc[15784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887728, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for HTTP/127.3.233.148@KRBTEST.COM
I20260619 16:48:48.992257 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:48.977994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.3.233.190:34343)
0619 16:48:48.978668 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:48.978672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:48.978684 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:48:48.978722 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:48.978725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:48.978763 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:48.978866 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:48.980021 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.980753 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.982049 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:48.982518 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:48.982640 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:48.987953 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:48.987957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:48.987959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:48.987991 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:48.989599 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:48.990193 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:48.990197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:48.990199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:48.990297 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:48.991267 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:48.991276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:48.991278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:48.991511 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:48.991674 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:48.991925 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:48:48.992060 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":116,"threads_started":1}
I20260619 16:48:48.996667 16774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:48:49.001454 16774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:48:49.027314 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.019848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.3.233.129:43975)
0619 16:48:49.020890 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:49.020896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:49.021303 (+ 407us) server_negotiation.cc:407] Connection header received
0619 16:48:49.021382 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:49.021387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:49.021559 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:49.021731 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:49.022261 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.023479 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.025711 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.026430 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.026556 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.026732 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:49.027037 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:48:49.027147 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":122,"threads_started":1}
I20260619 16:48:49.028398 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.018988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:43975 (local address 127.0.0.1:55812)
0619 16:48:49.020959 (+ 1971us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:49.021018 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:49.021161 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:49.021954 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:49.021961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:49.021985 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:49.022120 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.022128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.023697 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.023704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.025566 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.025579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.026696 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.026700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.026817 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.026859 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:48:49.028058 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0619 16:48:49.028209 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1766,"mutex_wait_us":327,"thread_start_us":97,"threads_started":1}
I20260619 16:48:49.032932 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.019381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:43695 (local address 127.0.0.1:48538)
0619 16:48:49.022655 (+ 3274us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:49.022738 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:49.023120 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:49.024588 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:49.024591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:49.024609 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:49.025149 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.025161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.027823 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.027827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.030219 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.030231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.032527 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.032530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.032628 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.032651 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:48:49.032719 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:48:49.032793 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3041,"thread_start_us":92,"threads_started":1}
I20260619 16:48:49.033435 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.019834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.3.233.131:43695)
0619 16:48:49.020673 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:49.020681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:49.023012 (+ 2331us) server_negotiation.cc:407] Connection header received
0619 16:48:49.023398 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:49.023406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:49.023886 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:49.024382 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:49.025608 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.027693 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.030741 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.032183 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.032308 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.032454 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:49.033193 (+ 739us) server_negotiation.cc:299] Negotiation successful
0619 16:48:49.033299 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":133,"threads_started":1}
I20260619 16:48:49.033253 16893 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2e0987c5064744acc3bcfc56c8c2c6 (DEFAULT_TABLE table=test-table [id=3f10c5dc3f654c5881ed5e084d2c9bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:49.034551 16893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2e0987c5064744acc3bcfc56c8c2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:49.036511 17162 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2e0987c5064744acc3bcfc56c8c2c6 (DEFAULT_TABLE table=test-table [id=3f10c5dc3f654c5881ed5e084d2c9bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:49.037586 17162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2e0987c5064744acc3bcfc56c8c2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:49.048228 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.019094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:37167 (local address 127.0.0.1:59988)
0619 16:48:49.021034 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:49.021048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:49.021622 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:49.028548 (+ 6926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:49.028552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:49.028570 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:49.028654 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.028660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.038979 (+ 10319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.038984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.041185 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.041199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.047702 (+ 6503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.047706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.047825 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.047852 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:48:49.047947 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:48:49.048054 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"thread_start_us":121,"threads_started":1}
I20260619 16:48:49.048843 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.019834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.3.233.130:37167)
0619 16:48:49.025020 (+ 5186us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:49.025027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:49.025057 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:48:49.026831 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:49.026838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:49.028212 (+ 1374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:49.028371 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:49.028779 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.029887 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.046284 (+ 16397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.047289 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.047464 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.047631 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:49.048556 (+ 925us) server_negotiation.cc:299] Negotiation successful
0619 16:48:49.048698 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5061,"thread_start_us":67,"threads_started":1}
I20260619 16:48:49.051349 17279 tablet_bootstrap.cc:492] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Bootstrap starting.
I20260619 16:48:49.052217 17044 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2e0987c5064744acc3bcfc56c8c2c6 (DEFAULT_TABLE table=test-table [id=3f10c5dc3f654c5881ed5e084d2c9bc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:49.053392 17044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2e0987c5064744acc3bcfc56c8c2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:49.053872 17279 tablet_bootstrap.cc:654] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:49.054792 17279 log.cc:824] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:49.068434 17281 tablet_bootstrap.cc:492] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Bootstrap starting.
I20260619 16:48:49.069041 17279 tablet_bootstrap.cc:492] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: No bootstrap required, opened a new log
I20260619 16:48:49.069273 17279 ts_tablet_manager.cc:1397] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260619 16:48:49.070609 17281 tablet_bootstrap.cc:654] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:49.071306 17282 tablet_bootstrap.cc:492] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848: Bootstrap starting.
I20260619 16:48:49.071540 17281 log.cc:824] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:49.073814 17282 tablet_bootstrap.cc:654] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:49.074466 17281 tablet_bootstrap.cc:492] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: No bootstrap required, opened a new log
I20260619 16:48:49.074671 17281 ts_tablet_manager.cc:1397] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:48:49.074502 17279 raft_consensus.cc:348] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.074816 17279 raft_consensus.cc:374] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:49.074882 17279 raft_consensus.cc:729] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efe8c4f7b3e491a8273b03af60c2401, State: Initialized, Role: FOLLOWER
I20260619 16:48:49.075397 17282 log.cc:824] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:49.075495 17279 consensus_queue.cc:260] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.076668 17279 ts_tablet_manager.cc:1428] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260619 16:48:49.077967 17282 tablet_bootstrap.cc:492] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848: No bootstrap required, opened a new log
I20260619 16:48:49.078171 17282 ts_tablet_manager.cc:1397] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:48:49.079444 17246 heartbeater.cc:503] Master 127.3.233.190:34343 was elected leader, sending a full tablet report...
I20260619 16:48:49.080307 17281 raft_consensus.cc:348] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.080610 17281 raft_consensus.cc:374] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:49.080662 17281 raft_consensus.cc:729] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0351a65e48741d4a64d17a7e6278ed9, State: Initialized, Role: FOLLOWER
I20260619 16:48:49.081179 17281 consensus_queue.cc:260] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.082306 17281 ts_tablet_manager.cc:1428] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260619 16:48:49.082696 17113 heartbeater.cc:503] Master 127.3.233.190:34343 was elected leader, sending a full tablet report...
I20260619 16:48:49.082979 17282 raft_consensus.cc:348] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.083324 17282 raft_consensus.cc:374] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:49.083439 17282 raft_consensus.cc:729] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed0ebd2425c423ebd82e4c31f8f2848, State: Initialized, Role: FOLLOWER
I20260619 16:48:49.084040 17282 consensus_queue.cc:260] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.085321 17282 ts_tablet_manager.cc:1428] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:48:49.085691 16978 heartbeater.cc:503] Master 127.3.233.190:34343 was elected leader, sending a full tablet report...
I20260619 16:48:49.129294 17287 raft_consensus.cc:482] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:48:49.129609 17287 raft_consensus.cc:504] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.130915 17287 leader_election.cc:288] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0351a65e48741d4a64d17a7e6278ed9 (127.3.233.130:37167), 7efe8c4f7b3e491a8273b03af60c2401 (127.3.233.131:43695)
I20260619 16:48:49.137486 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.131761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:56453 (local address 127.3.233.131:43695)
0619 16:48:49.131894 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:49.131899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:49.131915 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:48:49.132000 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:49.132005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:49.132062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:49.132155 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:49.133446 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.134069 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.136035 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.136657 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.136759 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.136816 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:49.137250 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 16:48:49.137336 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:48:49.137528 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.131413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:43899 (local address 127.3.233.130:37167)
0619 16:48:49.131574 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:49.131579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:49.131787 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:48:49.132018 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:49.132024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:49.132087 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:49.132138 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:49.132576 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.133293 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.136032 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.136656 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.136759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.136816 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:49.137249 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 16:48:49.137336 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:48:49.137907 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.131271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:43695 (local address 127.3.233.129:56453)
0619 16:48:49.131715 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:49.131731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:49.131851 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:49.132233 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:49.132238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:49.132265 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:49.132421 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.132443 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.134199 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.134203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.135868 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.135881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.136919 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.136923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.137075 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.137123 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:48:49.137655 (+ 532us) client_negotiation.cc:240] Negotiation successful
0619 16:48:49.137748 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":246,"thread_start_us":145,"threads_started":1}
I20260619 16:48:49.137938 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.131242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:37167 (local address 127.3.233.129:43899)
0619 16:48:49.131683 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:49.131699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:49.131857 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:49.132327 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:49.132329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:49.132343 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:49.132449 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.132454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.134221 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.134223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.135868 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:49.135881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.136919 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.136923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:49.137123 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.137140 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:48:49.137763 (+ 623us) client_negotiation.cc:240] Negotiation successful
0619 16:48:49.137818 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":105,"spinlock_wait_cycles":13184,"thread_start_us":154,"threads_started":1}
I20260619 16:48:49.138622 17067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2e0987c5064744acc3bcfc56c8c2c6" candidate_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0351a65e48741d4a64d17a7e6278ed9" is_pre_election: true
I20260619 16:48:49.138635 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2e0987c5064744acc3bcfc56c8c2c6" candidate_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7efe8c4f7b3e491a8273b03af60c2401" is_pre_election: true
I20260619 16:48:49.139031 17182 raft_consensus.cc:2454] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed0ebd2425c423ebd82e4c31f8f2848 in term 0.
I20260619 16:48:49.139032 17067 raft_consensus.cc:2454] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed0ebd2425c423ebd82e4c31f8f2848 in term 0.
I20260619 16:48:49.139564 16867 leader_election.cc:302] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ed0ebd2425c423ebd82e4c31f8f2848, 7efe8c4f7b3e491a8273b03af60c2401, b0351a65e48741d4a64d17a7e6278ed9; no voters:
I20260619 16:48:49.139887 17287 raft_consensus.cc:2781] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:48:49.139984 17287 raft_consensus.cc:482] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:48:49.140026 17287 raft_consensus.cc:3037] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:49.141391 17287 raft_consensus.cc:504] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.142195 17067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2e0987c5064744acc3bcfc56c8c2c6" candidate_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0351a65e48741d4a64d17a7e6278ed9"
I20260619 16:48:49.142256 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2e0987c5064744acc3bcfc56c8c2c6" candidate_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7efe8c4f7b3e491a8273b03af60c2401"
I20260619 16:48:49.142318 17067 raft_consensus.cc:3037] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:49.142370 17182 raft_consensus.cc:3037] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:49.143616 17067 raft_consensus.cc:2454] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed0ebd2425c423ebd82e4c31f8f2848 in term 1.
I20260619 16:48:49.143676 17182 raft_consensus.cc:2454] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed0ebd2425c423ebd82e4c31f8f2848 in term 1.
I20260619 16:48:49.143934 16867 leader_election.cc:302] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed0ebd2425c423ebd82e4c31f8f2848, b0351a65e48741d4a64d17a7e6278ed9; no voters:
I20260619 16:48:49.144156 17287 leader_election.cc:288] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [CANDIDATE]: Term 1 election: Requested vote from peers b0351a65e48741d4a64d17a7e6278ed9 (127.3.233.130:37167), 7efe8c4f7b3e491a8273b03af60c2401 (127.3.233.131:43695)
I20260619 16:48:49.144392 17290 raft_consensus.cc:2781] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:49.144498 17290 raft_consensus.cc:686] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed0ebd2425c423ebd82e4c31f8f2848, State: Running, Role: LEADER
I20260619 16:48:49.144891 17290 consensus_queue.cc:237] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } }
I20260619 16:48:49.148773 16774 catalog_manager.cc:5495] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ed0ebd2425c423ebd82e4c31f8f2848 (127.3.233.129). New cstate: current_term: 1 leader_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed0ebd2425c423ebd82e4c31f8f2848" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 43975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 } health_report { overall_health: UNKNOWN } } }
W20260619 16:48:49.158125 16979 tablet.cc:2369] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:48:49.190984 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.185063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.3.233.129:43975)
0619 16:48:49.185207 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:49.185211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:49.185225 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:48:49.185317 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:49.185320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:49.185395 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:49.185508 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:49.185944 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.186760 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.188537 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:49.188878 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:49.189014 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:49.189494 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:49.190114 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:49.190239 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:49.190696 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 16:48:49.190771 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 16:48:49.191951 17114 tablet.cc:2369] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:48:49.198462 17182 raft_consensus.cc:1261] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 1 FOLLOWER]: Refusing update from remote peer 1ed0ebd2425c423ebd82e4c31f8f2848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:49.198540 17067 raft_consensus.cc:1261] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 1 FOLLOWER]: Refusing update from remote peer 1ed0ebd2425c423ebd82e4c31f8f2848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:49.199297 17290 consensus_queue.cc:1035] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7efe8c4f7b3e491a8273b03af60c2401" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:49.199546 17287 consensus_queue.cc:1035] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 1ed0ebd2425c423ebd82e4c31f8f2848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0351a65e48741d4a64d17a7e6278ed9" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:49.215286 17294 mvcc.cc:205] Tried to move back new op lower bound from 7298612138788356096 to 7298612138580729856. Current Snapshot: MvccSnapshot[applied={T|T < 7298612138788356096}]
I20260619 16:48:49.218814 17295 mvcc.cc:205] Tried to move back new op lower bound from 7298612138788356096 to 7298612138580729856. Current Snapshot: MvccSnapshot[applied={T|T < 7298612138788356096}]
I20260619 16:48:49.223090 16780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:49.223281 16780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:49.225975 16780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45b28ac44c3b4d8493f31595b9a81ee8: Sending DeleteTablet for 3 replicas of tablet 8d2e0987c5064744acc3bcfc56c8c2c6
I20260619 16:48:49.227088 16893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2e0987c5064744acc3bcfc56c8c2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:49 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:55812
I20260619 16:48:49.227604 17044 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2e0987c5064744acc3bcfc56c8c2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:49 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:59988
I20260619 16:48:49.228206 17162 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2e0987c5064744acc3bcfc56c8c2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:49 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:48538
I20260619 16:48:49.228263 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 16818
I20260619 16:48:49.232627 17304 tablet_replica.cc:330] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: stopping tablet replica
I20260619 16:48:49.232980 17304 raft_consensus.cc:2229] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:49.233325 17304 raft_consensus.cc:2258] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:49.235153 17304 ts_tablet_manager.cc:1905] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:49.238199 17304 ts_tablet_manager.cc:1918] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:49.238322 17304 log.cc:1196] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/8d2e0987c5064744acc3bcfc56c8c2c6
I20260619 16:48:49.238672 17304 ts_tablet_manager.cc:1939] T 8d2e0987c5064744acc3bcfc56c8c2c6 P 7efe8c4f7b3e491a8273b03af60c2401: Deleting consensus metadata
I20260619 16:48:49.240151 16765 catalog_manager.cc:4841] TS 7efe8c4f7b3e491a8273b03af60c2401 (127.3.233.131:43695): tablet 8d2e0987c5064744acc3bcfc56c8c2c6 (table test-table [id=3f10c5dc3f654c5881ed5e084d2c9bc0]) successfully deleted
I20260619 16:48:49.240463 17303 tablet_replica.cc:330] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: stopping tablet replica
I20260619 16:48:49.240833 17303 raft_consensus.cc:2229] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:49.241186 17303 raft_consensus.cc:2258] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:49.244656 17303 ts_tablet_manager.cc:1905] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:49.247912 17303 ts_tablet_manager.cc:1918] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:49.248122 17303 log.cc:1196] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/8d2e0987c5064744acc3bcfc56c8c2c6
W20260619 16:48:49.248298 16764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:43975 (error 108)
I20260619 16:48:49.248625 17303 ts_tablet_manager.cc:1939] T 8d2e0987c5064744acc3bcfc56c8c2c6 P b0351a65e48741d4a64d17a7e6278ed9: Deleting consensus metadata
I20260619 16:48:49.249497 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 16980
I20260619 16:48:49.252560 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.252200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:43975 (local address 127.0.0.1:55834)
0619 16:48:49.252386 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:49.252456 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:43975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260619 16:48:49.252844 16764 catalog_manager.cc:4584] TS 1ed0ebd2425c423ebd82e4c31f8f2848 (127.3.233.129:43975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d2e0987c5064744acc3bcfc56c8c2c6: Network error: Client connection negotiation failed: client connection to 127.3.233.129:43975: connect: Connection refused (error 111)
I20260619 16:48:49.258100 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17116
I20260619 16:48:49.260298 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:49.259960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:37167 (local address 127.0.0.1:59990)
0619 16:48:49.260132 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:49.260201 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.130:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:48:49.265499 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 16747
I20260619 16:48:49.272766 4006 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:48:49.618338 4006 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:48:49.618520 4006 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:48:49.964316 4006 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:48:49.964 UTC [15814] LOG: received smart shutdown request
2026-06-19 16:48:49.968 UTC [15814] LOG: background worker "logical replication launcher" (PID 15823) exited with exit code 1
2026-06-19 16:48:49.968 UTC [15818] LOG: shutting down
2026-06-19 16:48:50.147 UTC [15814] LOG: database system is shut down
2026-06-19T16:48:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49379 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:50 dist-test-slave-cgwx krb5kdc[17314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:50 dist-test-slave-cgwx krb5kdc[17314](info): set up 2 sockets
Jun 19 16:48:50 dist-test-slave-cgwx krb5kdc[17314](info): commencing operation
krb5kdc: starting...
W20260619 16:48:51.339067 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:48:51 dist-test-slave-cgwx krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:48:51.465618 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:43787
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:43787,127.3.233.189:35627,127.3.233.188:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:51.575997 17327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:51.576347 17327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:51.576388 17327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:51.580619 17327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:51.580708 17327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:51.580740 17327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:51.580773 17327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:51.580801 17327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:51.586181 17327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:43787,127.3.233.189:35627,127.3.233.188:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:51.587299 17327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:51.588267 17327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:51.597115 17332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:51.597118 17333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:51.597074 17335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:51.597183 17327 server_base.cc:1034] running on GCE node
I20260619 16:48:51.597841 17327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:51.599007 17327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:51.600200 17327 hybrid_clock.cc:648] HybridClock initialized: now 1781887731600185 us; error 27 us; skew 500 ppm
Jun 19 16:48:51 dist-test-slave-cgwx krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:51.603048 17327 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:48:51.603924 17327 webserver.cc:460] Webserver started at http://127.3.233.190:37261/ using document root <none> and password file <none>
I20260619 16:48:51.604449 17327 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:51.604507 17327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:51.604676 17327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:51.606684 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "4d9da72870374316a27c8f3c65035b6d"
format_stamp: "Formatted at 2026-06-19 16:48:51 on dist-test-slave-cgwx"
server_key: "8ce47f517de75d4d8c8f9a454c73508e"
server_key_iv: "1b878ae501594ea6e4855fe35f73f52a"
server_key_version: "encryptionkey@0"
I20260619 16:48:51.607313 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "4d9da72870374316a27c8f3c65035b6d"
format_stamp: "Formatted at 2026-06-19 16:48:51 on dist-test-slave-cgwx"
server_key: "8ce47f517de75d4d8c8f9a454c73508e"
server_key_iv: "1b878ae501594ea6e4855fe35f73f52a"
server_key_version: "encryptionkey@0"
I20260619 16:48:51.611380 17327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:48:51.614075 17342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:51.615774 17327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:51.615900 17327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "4d9da72870374316a27c8f3c65035b6d"
format_stamp: "Formatted at 2026-06-19 16:48:51 on dist-test-slave-cgwx"
server_key: "8ce47f517de75d4d8c8f9a454c73508e"
server_key_iv: "1b878ae501594ea6e4855fe35f73f52a"
server_key_version: "encryptionkey@0"
I20260619 16:48:51.616017 17327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:51.628285 17327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:51.631862 17327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:51.632042 17327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:51.639758 17327 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:43787
I20260619 16:48:51.639846 17394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:43787 every 8 connection(s)
I20260619 16:48:51.640919 17327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:51.641932 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17327
I20260619 16:48:51.642079 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:48:51.642385 4006 external_mini_cluster.cc:1428] Setting key a6ce557b57cd7767a6a5b06f66597aa4
I20260619 16:48:51.644660 17395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:51.648535 17395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.233.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.189@KRBTEST.COM" created.
Jun 19 16:48:51 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:51.665421 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.651082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43787 (local address 127.0.0.1:58588)
0619 16:48:51.651630 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.651679 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:51.652636 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:51.653365 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:51.653370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:51.653822 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:51.654143 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.654163 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.655452 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.655457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.656644 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.656654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.657579 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.657582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.657734 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.658394 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:51.658410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:51.659831 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:51.662294 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.662301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.662310 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.662522 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.663009 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.663012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.663013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.663135 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.663790 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:51.663795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:51.663948 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 16:48:51.664160 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 16:48:51.664570 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":97,"thread_start_us":168,"threads_started":1}
I20260619 16:48:51.665395 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.651082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.3.233.190:43787)
0619 16:48:51.652023 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:51.652034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:51.652074 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:48:51.652942 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:51.652955 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:51.653183 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:51.653370 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:51.654331 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.655320 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.656769 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.657435 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.657618 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.659983 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:51.659990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:51.660004 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:51.660031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:51.662182 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.662646 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.662652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.662653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.662780 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.663227 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.663230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.663232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.663683 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:51.663800 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:51.664264 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 16:48:51.664498 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":826,"spinlock_wait_cycles":56064,"thread_start_us":168,"threads_started":1}
I20260619 16:48:51.668001 17395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:51.668799 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.668503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41480)
0619 16:48:51.668636 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.668740 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:48:51.669032 17344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab.
Entry for principal kudu/127.3.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab.
I20260619 16:48:51.669962 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.669679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41490)
0619 16:48:51.669828 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.669884 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 16:48:51.670106 17395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:35627: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
I20260619 16:48:51.670800 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:35627
--webserver_interface=127.3.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab
--principal=kudu/127.3.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:43787,127.3.233.189:35627,127.3.233.188:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:51.719389 17395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } attempt: 1
I20260619 16:48:51.720324 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.719973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41502)
0619 16:48:51.720137 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.720230 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:48:51.721195 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.720964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41508)
0619 16:48:51.721074 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.721130 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:48:51.721369 17395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:35627: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
W20260619 16:48:51.788971 17402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:51.789239 17402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:51.789280 17402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:48:51.789574 17395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } attempt: 2
I20260619 16:48:51.790445 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.790110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41512)
0619 16:48:51.790278 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.790354 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:48:51.791416 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.791170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41514)
0619 16:48:51.791293 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.791348 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:48:51.791603 17395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:35627: Network error: Client connection negotiation failed: client connection to 127.3.233.189:35627: connect: Connection refused (error 111)
W20260619 16:48:51.794162 17402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:51.794255 17402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:51.794301 17402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:51.794337 17402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:51.794368 17402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:51.800238 17402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:43787,127.3.233.189:35627,127.3.233.188:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab
--principal=kudu/127.3.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:51.801373 17402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:51.802428 17402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:51.811044 17407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:51.811093 17408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:51.811044 17410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:51.811334 17402 server_base.cc:1034] running on GCE node
I20260619 16:48:51.812027 17402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:51.813293 17402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:51.814532 17402 hybrid_clock.cc:648] HybridClock initialized: now 1781887731814522 us; error 28 us; skew 500 ppm
Jun 19 16:48:51 dist-test-slave-cgwx krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:51.817646 17402 init.cc:377] Logged in from keytab as kudu/127.3.233.189@KRBTEST.COM (short username kudu)
I20260619 16:48:51.818624 17402 webserver.cc:460] Webserver started at http://127.3.233.189:44695/ using document root <none> and password file <none>
I20260619 16:48:51.819254 17402 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:51.819347 17402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:51.819609 17402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:51.821729 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data/instance:
uuid: "8490169139aa4fd2b558dd47061c9215"
format_stamp: "Formatted at 2026-06-19 16:48:51 on dist-test-slave-cgwx"
server_key: "f7a6e3093b5d313cac13504f558b6c34"
server_key_iv: "588f6a33955da2278007b111e3429da7"
server_key_version: "encryptionkey@0"
I20260619 16:48:51.822300 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/wal/instance:
uuid: "8490169139aa4fd2b558dd47061c9215"
format_stamp: "Formatted at 2026-06-19 16:48:51 on dist-test-slave-cgwx"
server_key: "f7a6e3093b5d313cac13504f558b6c34"
server_key_iv: "588f6a33955da2278007b111e3429da7"
server_key_version: "encryptionkey@0"
I20260619 16:48:51.826171 17402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 16:48:51.828748 17417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:51.830044 17402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:48:51.830193 17402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/wal
uuid: "8490169139aa4fd2b558dd47061c9215"
format_stamp: "Formatted at 2026-06-19 16:48:51 on dist-test-slave-cgwx"
server_key: "f7a6e3093b5d313cac13504f558b6c34"
server_key_iv: "588f6a33955da2278007b111e3429da7"
server_key_version: "encryptionkey@0"
I20260619 16:48:51.830310 17402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:51.852433 17402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:51.856123 17402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:51.856348 17402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:51.864013 17402 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.189:35627
I20260619 16:48:51.864029 17469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.189:35627 every 8 connection(s)
I20260619 16:48:51.865162 17402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
I20260619 16:48:51.867901 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17402
I20260619 16:48:51.868094 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-1/wal/instance
I20260619 16:48:51.868216 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:51.868486 4006 external_mini_cluster.cc:1428] Setting key dd8cc92311771b1686397a657fa1461e
I20260619 16:48:51.872804 17470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.233.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.188@KRBTEST.COM" created.
I20260619 16:48:51.882795 17395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } attempt: 3
Jun 19 16:48:51 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:51.889503 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.874963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.3.233.190:43787)
0619 16:48:51.875128 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:51.875132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:51.876277 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 16:48:51.877403 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:51.877407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:51.877486 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:51.877590 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:51.878784 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.879637 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.880843 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.881096 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.881214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.884528 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:51.884535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:51.884539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:51.884576 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:51.886609 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.887266 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.887272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.887275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.887369 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.887849 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.887853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.887855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.888050 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:51.888129 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:51.889268 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0619 16:48:51.889370 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:48:51.889991 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.875475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43787 (local address 127.0.0.1:58592)
0619 16:48:51.876086 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.876145 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:51.877170 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:51.877770 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:51.877777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:51.878129 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:51.878628 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.878641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.879782 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.879785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.880695 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.880706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.881262 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.881266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.881448 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.882206 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:51.882227 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:51.884299 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:51.886718 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.886725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.886741 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.887149 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.887489 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.887492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.887494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.887703 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.888156 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:51.888161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:51.888465 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 16:48:51.888691 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 16:48:51.888969 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":162,"threads_started":1}
Jun 19 16:48:51 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:48:51.891578 17470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:51.895747 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.883292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41520)
0619 16:48:51.883431 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.883454 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:51.883534 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:51.884452 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:51.884455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:51.884656 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:51.884752 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.884757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.885926 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.885929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.886470 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.886478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.887313 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.887317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.887443 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.888172 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:51.888181 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:51.891208 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:51.893775 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.893778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.893779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.894052 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.894370 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.894374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.894376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.894477 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.895220 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:51.895223 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:51.895334 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 16:48:51.895420 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:48:51.895597 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:48:51.896046 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.883708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.3.233.189:35627)
0619 16:48:51.884029 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:51.884040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:51.884061 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:48:51.884100 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:51.884104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:51.884283 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:51.884463 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:51.884867 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.885797 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.886591 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.887175 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.887309 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.891356 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:51.891364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:51.891375 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:51.891397 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:51.893634 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.894152 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.894166 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.894168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.894272 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.894601 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.894604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.894605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.895105 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:51.895257 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:51.895599 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:48:51.895760 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":122,"threads_started":1}
Jun 19 16:48:51 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:48:51.898252 17395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab.
Entry for principal kudu/127.3.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab.
I20260619 16:48:51.899008 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.898731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47664)
0619 16:48:51.898850 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.898906 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:48:51.899955 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.899638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47668)
0619 16:48:51.899821 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.899875 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 16:48:51.900131 17395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:40075: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
I20260619 16:48:51.900229 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:40075
--webserver_interface=127.3.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab
--principal=kudu/127.3.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:43787,127.3.233.189:35627,127.3.233.188:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:51.902143 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.892149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41530)
0619 16:48:51.892261 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.892280 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:51.892371 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:51.892953 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:51.892958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:51.893133 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:51.893214 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.893219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.894275 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.894277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.895071 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:51.895126 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.895802 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.895807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:51.895905 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.896401 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:51.896412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:51.898412 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:51.900671 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.900677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.900679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.900870 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.901157 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:51.901160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:51.901162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:51.901242 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:51.901617 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:51.901619 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:51.901719 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 16:48:51.901825 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:48:51.901990 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 16:48:51.902284 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.892170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.3.233.189:35627)
0619 16:48:51.892555 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:51.892570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:51.892606 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:48:51.892647 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:51.892652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:51.892756 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:51.892870 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:51.893338 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.894167 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.895249 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:51.895655 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:51.895765 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:51.898617 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:51.898622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:51.898629 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:51.898649 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:51.900565 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.900965 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.900970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.900974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.901062 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:51.901329 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:51.901334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:51.901336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:51.901524 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:51.901630 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:51.901877 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 16:48:51.902092 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260619 16:48:51.903414 17470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:51.904165 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.903875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47676)
0619 16:48:51.903988 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.904095 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:48:51.904501 17418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
I20260619 16:48:51.905865 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.905650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47688)
0619 16:48:51.905758 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.905805 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:48:51.906026 17470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:40075: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
I20260619 16:48:51.931303 17395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } attempt: 1
I20260619 16:48:51.932296 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.931910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47702)
0619 16:48:51.932135 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.932213 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260619 16:48:51.933228 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.932925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47716)
0619 16:48:51.933058 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.933151 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:48:51.933465 17395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:40075: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
I20260619 16:48:51.955295 17470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } attempt: 1
I20260619 16:48:51.956311 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.955927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47726)
0619 16:48:51.956118 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.956199 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:48:51.957192 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:51.956935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47740)
0619 16:48:51.957053 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:51.957114 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:48:51.957348 17470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:40075: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
I20260619 16:48:52.008682 17395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } attempt: 2
I20260619 16:48:52.009538 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.009233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47750)
0619 16:48:52.009382 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.009463 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:48:52.010298 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.010089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47754)
0619 16:48:52.010197 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.010247 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:48:52.010433 17395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:40075: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
W20260619 16:48:52.017896 17477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:52.018206 17477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:52.018311 17477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:52.022737 17477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:52.022837 17477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:52.022881 17477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:52.022970 17477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:52.023034 17477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:52.025550 17470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } attempt: 2
I20260619 16:48:52.026502 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.026172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47766)
0619 16:48:52.026332 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.026402 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:48:52.027314 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.027093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47782)
0619 16:48:52.027201 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.027255 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:48:52.027462 17470 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:40075: Network error: Client connection negotiation failed: client connection to 127.3.233.188:40075: connect: Connection refused (error 111)
I20260619 16:48:52.029227 17477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:43787,127.3.233.189:35627,127.3.233.188:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab
--principal=kudu/127.3.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:52.030339 17477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:52.031451 17477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:52.040015 17484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:52.040061 17483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:52.040153 17477 server_base.cc:1034] running on GCE node
W20260619 16:48:52.040361 17486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:52.040939 17477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:52.042066 17477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:52.043279 17477 hybrid_clock.cc:648] HybridClock initialized: now 1781887732043261 us; error 32 us; skew 500 ppm
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887732, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:52.046414 17477 init.cc:377] Logged in from keytab as kudu/127.3.233.188@KRBTEST.COM (short username kudu)
I20260619 16:48:52.047415 17477 webserver.cc:460] Webserver started at http://127.3.233.188:40307/ using document root <none> and password file <none>
I20260619 16:48:52.048002 17477 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:52.048092 17477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:52.048321 17477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:52.050472 17477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data/instance:
uuid: "f030b6c96b9342868276626634db752a"
format_stamp: "Formatted at 2026-06-19 16:48:52 on dist-test-slave-cgwx"
server_key: "8a4f12e45ebf8938cea0ab72cd7fbc22"
server_key_iv: "72211629d1ec913ad907d965d6707a74"
server_key_version: "encryptionkey@0"
I20260619 16:48:52.051143 17477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/wal/instance:
uuid: "f030b6c96b9342868276626634db752a"
format_stamp: "Formatted at 2026-06-19 16:48:52 on dist-test-slave-cgwx"
server_key: "8a4f12e45ebf8938cea0ab72cd7fbc22"
server_key_iv: "72211629d1ec913ad907d965d6707a74"
server_key_version: "encryptionkey@0"
I20260619 16:48:52.055254 17477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:48:52.058054 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:52.059537 17477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:52.059695 17477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/wal
uuid: "f030b6c96b9342868276626634db752a"
format_stamp: "Formatted at 2026-06-19 16:48:52 on dist-test-slave-cgwx"
server_key: "8a4f12e45ebf8938cea0ab72cd7fbc22"
server_key_iv: "72211629d1ec913ad907d965d6707a74"
server_key_version: "encryptionkey@0"
I20260619 16:48:52.059819 17477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:52.085759 17477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:52.089413 17477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:52.089629 17477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:52.097050 17477 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.188:40075
I20260619 16:48:52.097083 17545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.188:40075 every 8 connection(s)
I20260619 16:48:52.098094 17477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
I20260619 16:48:52.101145 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:52.105144 17546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:52.107666 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17477
I20260619 16:48:52.107843 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887569513258-4006-0/minicluster-data/master-2/wal/instance
I20260619 16:48:52.108183 4006 external_mini_cluster.cc:1428] Setting key a06538ce7495a312e48a8158e7559608
I20260619 16:48:52.109567 17395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } attempt: 3
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887732, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:48:52.118677 17470 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } attempt: 3
I20260619 16:48:52.121937 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.107274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.3.233.190:43787)
0619 16:48:52.107498 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.107505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.108532 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 16:48:52.109839 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.109842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.109888 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.109975 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.111314 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.111830 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.113224 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.113624 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.113718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.117354 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.117357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.117359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.117379 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.119231 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.120029 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.120033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.120035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.120124 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.120611 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.120617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.120621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.120917 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.120992 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.121650 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.121771 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":102}
I20260619 16:48:52.122232 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.109971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47806)
0619 16:48:52.110140 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.110154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:52.110215 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:52.110659 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:52.110662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:52.110834 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:52.110968 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.110980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.112735 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.112739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.113558 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.113587 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.114201 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.114205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.114286 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.114796 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:52.114954 (+ 158us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:52.116514 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:52.120174 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.120177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.120180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.120456 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.120797 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.120801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.120804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.120893 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.121546 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:52.121552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:52.121667 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 16:48:52.121753 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:48:52.121890 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":30,"spinlock_wait_cycles":2944}
I20260619 16:48:52.124049 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.110055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.3.233.188:40075)
0619 16:48:52.110407 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.110415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.110439 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:48:52.110470 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.110473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.110514 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.111080 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.111165 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.112616 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.113704 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.114072 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.114495 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.116679 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.116685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.116718 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.116747 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.120082 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.120593 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.120599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.120601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.120685 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.121034 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.121037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.121039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.121419 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.122503 (+ 1084us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.122606 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.123256 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":528,"server-negotiator.queue_time_us":292,"spinlock_wait_cycles":5888,"thread_start_us":158,"threads_started":1}
I20260619 16:48:52.124047 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.107692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:43787 (local address 127.0.0.1:58594)
0619 16:48:52.108336 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.108412 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:52.109594 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:52.110239 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:52.110245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:52.110733 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:52.111127 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.111147 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.111941 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.111946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.113098 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.113108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.113869 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.113872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.114131 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.115042 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:52.115064 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:52.117200 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:52.119350 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.119359 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.119377 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.119917 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.120288 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.120293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.120297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.120494 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.121072 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:52.121078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:52.121395 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 16:48:52.122447 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0619 16:48:52.122833 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":218,"thread_start_us":179,"threads_started":1}
I20260619 16:48:52.124044 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.108824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.3.233.188:40075)
0619 16:48:52.109083 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.109097 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.109268 (+ 171us) server_negotiation.cc:407] Connection header received
0619 16:48:52.109617 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.109643 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.109889 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.110201 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.111326 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.112616 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.113621 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.114060 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.114181 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.117730 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.117737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.117743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.117772 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.121077 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.121575 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.121578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.121580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.121671 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.121986 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.121989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.121991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.122183 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.122850 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.122958 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.123071 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":312,"server-negotiator.queue_time_us":163,"spinlock_wait_cycles":56448,"thread_start_us":101,"threads_started":1}
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:48:52.126094 17546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:52.129467 17395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: Bootstrap starting.
I20260619 16:48:52.129634 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.119188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47816)
0619 16:48:52.119346 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.119365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:52.119457 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:52.119879 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:52.119882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:52.120361 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:52.120479 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.120488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.121608 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.121611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.122412 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.122422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.123027 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.123031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.123108 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.123582 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:52.123596 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:52.125008 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:52.128078 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.128082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.128084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.128362 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.128661 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.128666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.128668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.128755 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.129142 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:52.129145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:52.129241 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 16:48:52.129342 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:48:52.129466 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:48:52.130409 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.119270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.3.233.188:40075)
0619 16:48:52.119620 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.119632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.119648 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:48:52.119681 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.119685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.119728 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.120636 (+ 908us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.120742 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.121482 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.122542 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.122868 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.124151 (+ 1283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.125135 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.125139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.125141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.125165 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.127874 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.128473 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.128477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.128479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.128570 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.128850 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.128855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.128856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.129044 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.129152 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.130060 (+ 908us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.130193 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"spinlock_wait_cycles":1280,"thread_start_us":117,"threads_started":1}
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887732, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:48:52.132831 17395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:52.133729 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215: Bootstrap starting.
I20260619 16:48:52.133946 17395 log.cc:824] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:52.135649 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.126574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:35627 (local address 127.0.0.1:41534)
0619 16:48:52.126697 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.126713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:52.126783 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:52.127155 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:52.127157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:52.127360 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:52.127443 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.127448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.128347 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.128351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.129113 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.129122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.129878 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.129882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.129981 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.130479 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:52.130492 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:52.131961 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:52.134010 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.134013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.134016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.134245 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.134543 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.134545 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.134546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.134609 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.135157 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:52.135163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:52.135265 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 16:48:52.135371 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:48:52.135506 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:48:52.136137 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.126816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.3.233.189:35627)
0619 16:48:52.126955 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.126961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.126973 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:48:52.127007 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.127010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.127052 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.127131 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.127559 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.128225 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.129270 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.129618 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.129719 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.132111 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.132115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.132118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.132141 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.133843 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.134359 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.134363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.134365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.134441 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.134704 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.134709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.134711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.135000 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.135110 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.135873 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.135988 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:48:52.136497 17470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:52.136766 17546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:48:52.137203 17395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: No bootstrap required, opened a new log
I20260619 16:48:52.137364 17470 log.cc:824] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:52.139483 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215: No bootstrap required, opened a new log
I20260619 16:48:52.140710 17395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.140908 17395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:52.140950 17395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9da72870374316a27c8f3c65035b6d, State: Initialized, Role: FOLLOWER
I20260619 16:48:52.141381 17395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.142800 17395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:52.143442 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.131010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.3.233.190:43787)
0619 16:48:52.131133 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.131138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.131152 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:48:52.131188 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.131190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.131235 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.131308 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.132285 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.133666 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.135044 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.135361 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.135434 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.139543 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.139547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.139550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.139577 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.141522 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.142026 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.142031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.142033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.142124 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.142438 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.142442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.142444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.142684 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.142916 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.143148 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.143281 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:48:52.144076 17470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.144307 17470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:52.144357 17470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8490169139aa4fd2b558dd47061c9215, State: Initialized, Role: FOLLOWER
I20260619 16:48:52.145481 17470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.146885 17470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887732, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
Jun 19 16:48:52 dist-test-slave-cgwx krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:48:52.147878 17327 master_runner.cc:186] Error getting master registration for 127.3.233.189:35627: OK, instance_id { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" instance_seqno: 1781887731861563 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:48:52.152140 17327 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:48:52.154093 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.137281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:40075 (local address 127.0.0.1:47818)
0619 16:48:52.137596 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:52.137609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:52.137678 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:52.144741 (+ 7063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:52.144747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:52.144964 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:52.145089 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.145099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.146081 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.146084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.146812 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:52.146822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.147940 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.147943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:52.148022 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.148536 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:52.148549 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:52.150148 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:52.152627 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.152630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.152633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.152868 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.153134 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:52.153137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:52.153138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:52.153234 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:52.153647 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:52.153650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:52.153723 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 16:48:52.153804 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:48:52.153937 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260619 16:48:52.154670 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.137494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.3.233.188:40075)
0619 16:48:52.137942 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.137947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.137964 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:48:52.138016 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.138022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.138065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.144718 (+ 6653us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.145219 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.145970 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.146974 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.147823 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.148241 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.150308 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.150314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.150317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.150340 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.152519 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.152975 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.152978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.152979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.153050 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.153330 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.153332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.153334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.153544 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.154340 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.154409 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.154538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260619 16:48:52.156962 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.145098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.3.233.189:35627)
0619 16:48:52.145211 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.145215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.145227 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:48:52.145332 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.145335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.145377 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.145438 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.146298 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.147084 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.148548 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.148905 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.149010 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.152684 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.152690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.152692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.152718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.154411 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.155304 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.155308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.155310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.155390 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.155688 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.155691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.155693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.155889 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.156069 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.156702 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.156820 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:48:52.157577 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: Bootstrap starting.
I20260619 16:48:52.154503 17402 master_runner.cc:186] Error getting master registration for 127.3.233.188:40075: OK, instance_id { permanent_uuid: "f030b6c96b9342868276626634db752a" instance_seqno: 1781887732094763 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:48:52.158012 17402 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:48:52.159087 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.159224 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:48:52.159384 17566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:48:52.159439 17566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:48:52.160262 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:52.160352 17546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:52.161211 17546 log.cc:824] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: Log is configured to *not* fsync() on all Append() calls
W20260619 16:48:52.161410 17578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:48:52.161480 17578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:48:52.163137 17567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.163704 17567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:52.165656 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: No bootstrap required, opened a new log
I20260619 16:48:52.170622 17546 raft_consensus.cc:348] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.170897 17546 raft_consensus.cc:374] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:52.171025 17546 raft_consensus.cc:729] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f030b6c96b9342868276626634db752a, State: Initialized, Role: FOLLOWER
I20260619 16:48:52.171563 17546 consensus_queue.cc:260] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.172940 17546 sys_catalog.cc:564] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:52.180673 17477 master_runner.cc:263] No leader master found from master f030b6c96b9342868276626634db752a
I20260619 16:48:52.180764 17477 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:48:52.181335 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.163468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.3.233.190:43787)
0619 16:48:52.163590 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.163595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.171293 (+ 7698us) server_negotiation.cc:407] Connection header received
0619 16:48:52.171533 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.171538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.171599 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.171681 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.172832 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.173641 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.174871 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.175260 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.175340 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.177248 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.177252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.177255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.177286 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.179172 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.179700 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.179705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.179706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.179792 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.180138 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.180141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.180143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.180371 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.180498 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.181005 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.181143 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 16:48:52.184401 17600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:48:52.184508 17600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:48:52.187325 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.163737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.3.233.188:40075)
0619 16:48:52.163852 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.163858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.171728 (+ 7870us) server_negotiation.cc:407] Connection header received
0619 16:48:52.171898 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.171901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.171950 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.172019 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.179048 (+ 7029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.179813 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.180949 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.181266 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.181358 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.183515 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.183519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.183522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.183552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.185343 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.185919 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.185923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.185924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.186012 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.186314 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.186317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.186319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.186545 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.186656 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.187040 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.187168 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:48:52.189036 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.189239 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:52.199018 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:52.164308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.3.233.189:35627)
0619 16:48:52.164407 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:52.164411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:52.182156 (+ 17745us) server_negotiation.cc:407] Connection header received
0619 16:48:52.184076 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:52.184082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:52.184142 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:52.184198 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:52.186161 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.186946 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.188095 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:52.188415 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:52.188521 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:52.195067 (+ 6546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:52.195073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:52.195076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:52.195115 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:52.197034 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.197582 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.197587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.197591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.197679 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:52.197975 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:52.197978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:52.197980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:52.198211 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:52.198307 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:52.198683 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 16:48:52.198810 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:48:52.216883 17567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:48:52.217085 17567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.218160 17567 leader_election.cc:288] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d9da72870374316a27c8f3c65035b6d (127.3.233.190:43787), f030b6c96b9342868276626634db752a (127.3.233.188:40075)
I20260619 16:48:52.219129 17369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8490169139aa4fd2b558dd47061c9215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d9da72870374316a27c8f3c65035b6d" is_pre_election: true
I20260619 16:48:52.219480 17369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8490169139aa4fd2b558dd47061c9215 in term 0.
I20260619 16:48:52.219925 17520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8490169139aa4fd2b558dd47061c9215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030b6c96b9342868276626634db752a" is_pre_election: true
I20260619 16:48:52.220252 17520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8490169139aa4fd2b558dd47061c9215 in term 0.
I20260619 16:48:52.221146 17419 leader_election.cc:302] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d9da72870374316a27c8f3c65035b6d, 8490169139aa4fd2b558dd47061c9215; no voters:
I20260619 16:48:52.222105 17567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:48:52.222214 17567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:48:52.222263 17567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:52.223754 17567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.224848 17520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8490169139aa4fd2b558dd47061c9215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030b6c96b9342868276626634db752a"
I20260619 16:48:52.225028 17520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:52.226404 17520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8490169139aa4fd2b558dd47061c9215 in term 1.
I20260619 16:48:52.226778 17418 leader_election.cc:302] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8490169139aa4fd2b558dd47061c9215, f030b6c96b9342868276626634db752a; no voters:
I20260619 16:48:52.227200 17369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8490169139aa4fd2b558dd47061c9215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d9da72870374316a27c8f3c65035b6d"
I20260619 16:48:52.227346 17369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:52.228664 17369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8490169139aa4fd2b558dd47061c9215 in term 1.
I20260619 16:48:52.229048 17567 leader_election.cc:288] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [CANDIDATE]: Term 1 election: Requested vote from peers 4d9da72870374316a27c8f3c65035b6d (127.3.233.190:43787), f030b6c96b9342868276626634db752a (127.3.233.188:40075)
I20260619 16:48:52.229218 17567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:52.229317 17567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [term 1 LEADER]: Becoming Leader. State: Replica: 8490169139aa4fd2b558dd47061c9215, State: Running, Role: LEADER
I20260619 16:48:52.229699 17567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } }
I20260619 16:48:52.232004 17567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8490169139aa4fd2b558dd47061c9215. Latest consensus state: current_term: 1 leader_uuid: "8490169139aa4fd2b558dd47061c9215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.232132 17567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:52.232568 17604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:52.233683 17604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:52.255005 17520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [term 1 FOLLOWER]: Refusing update from remote peer 8490169139aa4fd2b558dd47061c9215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:52.255715 17602 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:52.259928 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8490169139aa4fd2b558dd47061c9215. Latest consensus state: current_term: 1 leader_uuid: "8490169139aa4fd2b558dd47061c9215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.260087 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:52.261333 17369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [term 1 FOLLOWER]: Refusing update from remote peer 8490169139aa4fd2b558dd47061c9215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:52.264846 17602 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:52.269428 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8490169139aa4fd2b558dd47061c9215. Latest consensus state: current_term: 1 leader_uuid: "8490169139aa4fd2b558dd47061c9215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.269588 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:52.275501 17602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8490169139aa4fd2b558dd47061c9215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.275652 17602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:52.276758 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8490169139aa4fd2b558dd47061c9215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.276901 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:52.279774 17605 mvcc.cc:205] Tried to move back new op lower bound from 7298612151292215296 to 7298612151217422336. Current Snapshot: MvccSnapshot[applied={T|T < 7298612151292215296}]
I20260619 16:48:52.282612 17567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8490169139aa4fd2b558dd47061c9215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.282742 17567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:52.283874 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8490169139aa4fd2b558dd47061c9215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9da72870374316a27c8f3c65035b6d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43787 } } peers { permanent_uuid: "8490169139aa4fd2b558dd47061c9215" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 35627 } } peers { permanent_uuid: "f030b6c96b9342868276626634db752a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 40075 } } }
I20260619 16:48:52.283996 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:48:52.285151 17608 mvcc.cc:205] Tried to move back new op lower bound from 7298612151292215296 to 7298612151217422336. Current Snapshot: MvccSnapshot[applied={T|T < 7298612151292215296}]
I20260619 16:48:52.285321 17606 mvcc.cc:205] Tried to move back new op lower bound from 7298612151292215296 to 7298612151217422336. Current Snapshot: MvccSnapshot[applied={T|T < 7298612151292215296}]
I20260619 16:48:52.288803 17604 catalog_manager.cc:1347] Generated new cluster ID: a1aa33011bd649babe640f8283aa8484
I20260619 16:48:52.288892 17604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:52.318569 17604 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:52.320096 17604 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:52.337085 17604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8490169139aa4fd2b558dd47061c9215: Generated new TSK 0
I20260619 16:48:52.337937 17604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:53.160771 17566 catalog_manager.cc:1259] Loaded cluster ID: a1aa33011bd649babe640f8283aa8484
I20260619 16:48:53.160882 17566 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: loading cluster ID for follower catalog manager: success
I20260619 16:48:53.162806 17566 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: acquiring CA information for follower catalog manager: success
I20260619 16:48:53.164289 17566 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d9da72870374316a27c8f3c65035b6d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:48:53.185987 17600 catalog_manager.cc:1259] Loaded cluster ID: a1aa33011bd649babe640f8283aa8484
I20260619 16:48:53.186079 17600 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: loading cluster ID for follower catalog manager: success
I20260619 16:48:53.187912 17600 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: acquiring CA information for follower catalog manager: success
I20260619 16:48:53.189152 17600 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f030b6c96b9342868276626634db752a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:48:53.408789 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17327
I20260619 16:48:53.415174 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17402
I20260619 16:48:53.421332 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17477
2026-06-19T16:48:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3147 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:53 dist-test-slave-cgwx krb5kdc[17620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:53 dist-test-slave-cgwx krb5kdc[17620](info): set up 2 sockets
Jun 19 16:48:53 dist-test-slave-cgwx krb5kdc[17620](info): commencing operation
krb5kdc: starting...
W20260619 16:48:54.487275 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:48:54 dist-test-slave-cgwx krb5kdc[17620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:48:54.606351 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:41781
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:54.716977 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:54.717303 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:54.717381 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:54.721403 17633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:54.721501 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:54.721540 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:54.721577 17633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:54.721611 17633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:54.726897 17633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:41781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:54.728034 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:54.729081 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:48:54.737860 17633 server_base.cc:1034] running on GCE node
W20260619 16:48:54.737792 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:54.737821 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:54.737803 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:54.738732 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:54.739965 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:54.741313 17633 hybrid_clock.cc:648] HybridClock initialized: now 1781887734741307 us; error 56 us; skew 500 ppm
Jun 19 16:48:54 dist-test-slave-cgwx krb5kdc[17620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887734, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:54.749313 17633 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:48:54.750451 17633 webserver.cc:460] Webserver started at http://127.3.233.190:46119/ using document root <none> and password file <none>
I20260619 16:48:54.751242 17633 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:54.751348 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:54.751612 17633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:54.753883 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "ccde455f80c64e90b3113c32ec148527"
format_stamp: "Formatted at 2026-06-19 16:48:54 on dist-test-slave-cgwx"
server_key: "fb1b004b5dc3a913fff8dfa01f84990d"
server_key_iv: "3bb858d57ee9a473e664a660c1da66d3"
server_key_version: "encryptionkey@0"
I20260619 16:48:54.754510 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "ccde455f80c64e90b3113c32ec148527"
format_stamp: "Formatted at 2026-06-19 16:48:54 on dist-test-slave-cgwx"
server_key: "fb1b004b5dc3a913fff8dfa01f84990d"
server_key_iv: "3bb858d57ee9a473e664a660c1da66d3"
server_key_version: "encryptionkey@0"
I20260619 16:48:54.758530 17633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:48:54.761271 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:54.762662 17633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:54.762823 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "ccde455f80c64e90b3113c32ec148527"
format_stamp: "Formatted at 2026-06-19 16:48:54 on dist-test-slave-cgwx"
server_key: "fb1b004b5dc3a913fff8dfa01f84990d"
server_key_iv: "3bb858d57ee9a473e664a660c1da66d3"
server_key_version: "encryptionkey@0"
I20260619 16:48:54.763018 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:54.875066 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:54.878496 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:54.878733 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:54.886775 17633 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:41781
I20260619 16:48:54.886801 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:41781 every 8 connection(s)
I20260619 16:48:54.887889 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:54.890727 17701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:54.894841 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17633
I20260619 16:48:54.894999 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:48:54.895275 4006 external_mini_cluster.cc:1428] Setting key d1312a6177e98339d5d2f58a35aeb327
I20260619 16:48:54.896616 17701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527: Bootstrap starting.
I20260619 16:48:54.898908 17701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:54.899724 17701 log.cc:824] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:54.901599 17701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527: No bootstrap required, opened a new log
Jun 19 16:48:54 dist-test-slave-cgwx krb5kdc[17620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:54.906037 17701 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde455f80c64e90b3113c32ec148527" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41781 } }
I20260619 16:48:54.906322 17701 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:54.906363 17701 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccde455f80c64e90b3113c32ec148527, State: Initialized, Role: FOLLOWER
I20260619 16:48:54.906840 17701 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde455f80c64e90b3113c32ec148527" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41781 } }
I20260619 16:48:54.907042 17701 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:54.907133 17701 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:54.907265 17701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:54.908412 17701 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde455f80c64e90b3113c32ec148527" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41781 } }
I20260619 16:48:54.908785 17701 leader_election.cc:302] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccde455f80c64e90b3113c32ec148527; no voters:
I20260619 16:48:54.909168 17701 leader_election.cc:288] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:54.909240 17706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:54.909685 17706 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [term 1 LEADER]: Becoming Leader. State: Replica: ccde455f80c64e90b3113c32ec148527, State: Running, Role: LEADER
I20260619 16:48:54.910095 17706 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde455f80c64e90b3113c32ec148527" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41781 } }
I20260619 16:48:54.910503 17701 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:54.910797 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:54.896663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.3.233.190:41781)
0619 16:48:54.897146 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:54.897161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:54.897198 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:48:54.897868 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:54.897886 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:54.898230 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:54.898593 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:54.899451 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:54.900499 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:54.901351 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:54.901888 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:54.902080 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:54.905099 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:54.905105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:54.905119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:54.905146 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:54.907207 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:54.907729 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:54.907734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:54.907739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:54.907854 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:54.908165 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:54.908168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:54.908169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:54.908655 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:54.908800 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:54.909498 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 16:48:54.909803 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":134,"threads_started":1}
I20260619 16:48:54.911356 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccde455f80c64e90b3113c32ec148527. Latest consensus state: current_term: 1 leader_uuid: "ccde455f80c64e90b3113c32ec148527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde455f80c64e90b3113c32ec148527" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41781 } } }
I20260619 16:48:54.911589 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:54.911762 17707 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccde455f80c64e90b3113c32ec148527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde455f80c64e90b3113c32ec148527" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 41781 } } }
I20260619 16:48:54.911857 17707 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:54.912111 17715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:54.914173 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:48:54.915362 17715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:54.921066 17715 catalog_manager.cc:1347] Generated new cluster ID: 52bdbfaf7cb24b5fa02805f6e917845b
I20260619 16:48:54.921144 17715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:54.934187 17715 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:54.935535 17715 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:54.942677 17715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccde455f80c64e90b3113c32ec148527: Generated new TSK 0
I20260619 16:48:54.943686 17715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:48:54.970530 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17633
2026-06-19T16:48:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1549 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:48:55 dist-test-slave-cgwx krb5kdc[17728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:48:55 dist-test-slave-cgwx krb5kdc[17728](info): set up 2 sockets
Jun 19 16:48:55 dist-test-slave-cgwx krb5kdc[17728](info): commencing operation
krb5kdc: starting...
W20260619 16:48:56.015275 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:48:56 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:48:56.139496 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34023
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:56.256765 17741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:56.257119 17741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:56.257210 17741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:56.261380 17741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:56.261492 17741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:56.261530 17741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:56.261569 17741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:56.261600 17741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:56.267473 17741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:56.268666 17741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:56.269775 17741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:56.278561 17746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:56.278541 17749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:56.278643 17741 server_base.cc:1034] running on GCE node
W20260619 16:48:56.278548 17747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:56.279579 17741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:56.280738 17741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:56.281945 17741 hybrid_clock.cc:648] HybridClock initialized: now 1781887736281929 us; error 32 us; skew 500 ppm
Jun 19 16:48:56 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:56.284940 17741 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:48:56.285935 17741 webserver.cc:460] Webserver started at http://127.3.233.190:44705/ using document root <none> and password file <none>
I20260619 16:48:56.286729 17741 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:56.286830 17741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:56.287107 17741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:56.289150 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "f73aff31c41749f5a1740f1ba1a79287"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "99999d84e7e59bc12d52cf6fc455a594"
server_key_iv: "9022900364f79811affcc8b399907587"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.289672 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "f73aff31c41749f5a1740f1ba1a79287"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "99999d84e7e59bc12d52cf6fc455a594"
server_key_iv: "9022900364f79811affcc8b399907587"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.293651 17741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:48:56.296231 17756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.297700 17741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:48:56.297875 17741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "f73aff31c41749f5a1740f1ba1a79287"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "99999d84e7e59bc12d52cf6fc455a594"
server_key_iv: "9022900364f79811affcc8b399907587"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.298003 17741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:56.312970 17741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:56.316412 17741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:56.316628 17741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:56.323714 17741 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34023
I20260619 16:48:56.324014 17808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34023 every 8 connection(s)
I20260619 16:48:56.324707 17741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:56.326099 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17741
I20260619 16:48:56.326259 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:48:56.326578 4006 external_mini_cluster.cc:1428] Setting key b3b3b7aecdcfb1eb0778e545ee7f8fbe
I20260619 16:48:56.327862 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:56.334180 17809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Bootstrap starting.
Jun 19 16:48:56 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:56.336922 17809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:56.337891 17809 log.cc:824] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:56.340240 17809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: No bootstrap required, opened a new log
I20260619 16:48:56.341818 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:56.327982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.3.233.190:34023)
0619 16:48:56.328412 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:56.328428 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:56.328463 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:48:56.329099 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:56.329116 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:56.329428 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:56.329707 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:56.330631 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.331855 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.333047 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.333560 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.333704 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:56.336463 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:56.336469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:56.336485 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:56.336514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:56.338531 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.339083 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.339088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.339094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.339293 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.339605 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.339608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.339610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.340093 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:56.340291 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:56.340592 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:48:56.340888 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":135,"threads_started":1}
I20260619 16:48:56.344559 17809 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:56.344801 17809 raft_consensus.cc:374] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:56.344846 17809 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73aff31c41749f5a1740f1ba1a79287, State: Initialized, Role: FOLLOWER
I20260619 16:48:56.345280 17809 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:56.345430 17809 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:56.345474 17809 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:56.345546 17809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:56.346486 17809 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:56.346788 17809 leader_election.cc:302] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73aff31c41749f5a1740f1ba1a79287; no voters:
I20260619 16:48:56.347108 17809 leader_election.cc:288] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:48:56.347384 17814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:56.347621 17814 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 LEADER]: Becoming Leader. State: Replica: f73aff31c41749f5a1740f1ba1a79287, State: Running, Role: LEADER
I20260619 16:48:56.348001 17814 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:56.348416 17809 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:56.349864 17815 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f73aff31c41749f5a1740f1ba1a79287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } } }
I20260619 16:48:56.349993 17815 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:56.350327 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73aff31c41749f5a1740f1ba1a79287. Latest consensus state: current_term: 1 leader_uuid: "f73aff31c41749f5a1740f1ba1a79287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } } }
I20260619 16:48:56.350414 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: This master's current role is: LEADER
W20260619 16:48:56.353551 17829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:48:56.353629 17829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:48:56.353703 17825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:56.354509 17825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:56.359722 17825 catalog_manager.cc:1347] Generated new cluster ID: 419a48323e4d427ab5ff31327bb1c857
I20260619 16:48:56.359784 17825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:56.383980 17825 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:48:56.385025 17825 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:56.391505 17825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Generated new TSK 0
I20260619 16:48:56.392295 17825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:48:56.426586 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34023
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:56.559940 17835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:56.560202 17835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:56.560246 17835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:56.564350 17835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:56.564404 17835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:56.564493 17835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:48:56.569684 17835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:56.570716 17835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:56.571751 17835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:48:56.581432 17835 server_base.cc:1034] running on GCE node
W20260619 16:48:56.581384 17843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:56.581400 17841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:56.581384 17840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:56.582082 17835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:56.582795 17835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:56.584074 17835 hybrid_clock.cc:648] HybridClock initialized: now 1781887736583989 us; error 38 us; skew 500 ppm
Jun 19 16:48:56 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:56.586910 17835 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:48:56.587846 17835 webserver.cc:460] Webserver started at http://127.3.233.129:42785/ using document root <none> and password file <none>
I20260619 16:48:56.588421 17835 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:56.588482 17835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:56.588725 17835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:56.590777 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "0b6002a1dc884bc7a35ce64f237862c3"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "ba38c04949cc6f508c77b3d9cab79432"
server_key_iv: "9b86760a2f4b91da068631c5f1c2c93d"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.591391 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "0b6002a1dc884bc7a35ce64f237862c3"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "ba38c04949cc6f508c77b3d9cab79432"
server_key_iv: "9b86760a2f4b91da068631c5f1c2c93d"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.595293 17835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:48:56.597932 17850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.599501 17835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:56.599645 17835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "0b6002a1dc884bc7a35ce64f237862c3"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "ba38c04949cc6f508c77b3d9cab79432"
server_key_iv: "9b86760a2f4b91da068631c5f1c2c93d"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.599757 17835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:56.618310 17835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:56.621788 17835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:56.622030 17835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:56.622797 17835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:56.623731 17835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:56.623803 17835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.623872 17835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:56.623920 17835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.633422 17835 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:44361
I20260619 16:48:56.633445 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:44361 every 8 connection(s)
I20260619 16:48:56.634372 17835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:48:56 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:56.644235 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17835
I20260619 16:48:56.644387 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:48:56.644697 4006 external_mini_cluster.cc:1428] Setting key 9012ea6363e6457aa65d99f3e09dbe18
I20260619 16:48:56.648658 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:56.636256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:60723 (local address 127.3.233.190:34023)
0619 16:48:56.636409 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:56.636414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:56.637675 (+ 1261us) server_negotiation.cc:407] Connection header received
0619 16:48:56.638501 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:56.638503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:56.638547 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:56.638623 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:56.639871 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.640422 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.641560 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.641827 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.641910 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:56.644859 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:56.644862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:56.644864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:56.644888 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:56.646472 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.647030 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.647034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.647036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.647123 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.647472 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.647475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.647477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.647694 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:56.647801 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:56.648293 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 16:48:56.648489 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:48:56.649561 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:56.636687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:60723)
0619 16:48:56.637510 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:56.637554 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:56.638325 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:56.638786 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:56.638793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:56.639260 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:56.639739 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:56.639752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.640528 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.640530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:56.641447 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:56.641456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.641985 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.641988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:56.642147 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:56.642709 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:56.642727 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:56.644650 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:56.646597 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:56.646603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:56.646618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:56.646881 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:56.647220 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:56.647225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:56.647227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:56.647371 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:56.647798 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:56.647804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:56.648054 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 16:48:56.648242 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 16:48:56.648621 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":136,"threads_started":1}
I20260619 16:48:56.651046 17964 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34023
I20260619 16:48:56.651320 17964 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:56.651965 17964 heartbeater.cc:511] Master 127.3.233.190:34023 requested a full tablet report, sending...
I20260619 16:48:56.653745 17773 ts_manager.cc:194] Registered new tserver with Master: 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361)
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:48:56.655210 17773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:60723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:48:56.673595 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34023
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:56.783433 17969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:56.783727 17969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:56.783810 17969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:56.788189 17969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:56.788256 17969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:56.788354 17969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:48:56.794119 17969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:56.795292 17969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:56.796375 17969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:56.806421 17975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:56.806533 17969 server_base.cc:1034] running on GCE node
W20260619 16:48:56.806435 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:56.806434 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:56.807202 17969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:56.807894 17969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:56.809098 17969 hybrid_clock.cc:648] HybridClock initialized: now 1781887736809059 us; error 50 us; skew 500 ppm
Jun 19 16:48:56 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:56.812001 17969 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:48:56.812944 17969 webserver.cc:460] Webserver started at http://127.3.233.130:42897/ using document root <none> and password file <none>
I20260619 16:48:56.813542 17969 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:56.813632 17969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:56.813870 17969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:56.816219 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "7935fdd500bffd1089bdaf0743dea55a"
server_key_iv: "6ee614b85bac2b5ac46519ae1e0079fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.816834 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "7935fdd500bffd1089bdaf0743dea55a"
server_key_iv: "6ee614b85bac2b5ac46519ae1e0079fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.820832 17969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:48:56.823338 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.824640 17969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:48:56.824787 17969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "7935fdd500bffd1089bdaf0743dea55a"
server_key_iv: "6ee614b85bac2b5ac46519ae1e0079fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:56.824911 17969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:56.843767 17969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:56.846756 17969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:56.847028 17969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:56.847647 17969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:56.848466 17969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:56.848544 17969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.848609 17969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:56.848655 17969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:56.857792 17969 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:34521
I20260619 16:48:56.857810 18097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:34521 every 8 connection(s)
I20260619 16:48:56.858698 17969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:56.860025 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 17969
I20260619 16:48:56.860163 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:48:56.860452 4006 external_mini_cluster.cc:1428] Setting key 531fd7ff2a95d73aa397852d69f48f70
Jun 19 16:48:56 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887736, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:48:56.874387 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:56.860750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:48951 (local address 127.3.233.190:34023)
0619 16:48:56.860890 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:56.860893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:56.862315 (+ 1422us) server_negotiation.cc:407] Connection header received
0619 16:48:56.863759 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:56.863765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:56.863832 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:56.863930 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:56.865230 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.865769 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.867115 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.867346 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.867460 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:56.870367 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:56.870371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:56.870373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:56.870397 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:56.872039 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.872636 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.872640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.872642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.872742 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:56.873119 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:56.873124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:56.873126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:56.873347 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:56.873451 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:56.873986 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 16:48:56.874132 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:48:56.875208 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:56.861249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:48951)
0619 16:48:56.862133 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:56.862195 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:56.863499 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:56.864166 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:56.864177 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:56.864558 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:56.865073 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:56.865086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.865911 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.865915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:56.866996 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:56.867006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:56.867516 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:56.867520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:56.867695 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:56.868361 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:56.868378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:56.870177 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:56.872174 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:56.872181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:56.872198 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:56.872523 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:56.872840 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:56.872844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:56.872863 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:56.873008 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:56.873463 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:56.873468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:56.873730 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 16:48:56.873900 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:48:56.874171 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":148,"threads_started":1}
I20260619 16:48:56.876605 18098 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34023
I20260619 16:48:56.876876 18098 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:56.877487 18098 heartbeater.cc:511] Master 127.3.233.190:34023 requested a full tablet report, sending...
I20260619 16:48:56.878754 17773 ts_manager.cc:194] Registered new tserver with Master: ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521)
I20260619 16:48:56.879606 17773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:48951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:48:56.890018 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34023
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:57.002741 18103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:57.003115 18103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:57.003217 18103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:57.007507 18103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:57.007599 18103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:57.007694 18103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:48:57.013266 18103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:57.014443 18103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:57.015585 18103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:57.026072 18111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.026117 18109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.026072 18108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:57.026371 18103 server_base.cc:1034] running on GCE node
I20260619 16:48:57.026973 18103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:57.027668 18103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:57.028859 18103 hybrid_clock.cc:648] HybridClock initialized: now 1781887737028845 us; error 33 us; skew 500 ppm
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:57.031841 18103 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:48:57.032784 18103 webserver.cc:460] Webserver started at http://127.3.233.131:46047/ using document root <none> and password file <none>
I20260619 16:48:57.033382 18103 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:57.033478 18103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:57.033727 18103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:48:57.035928 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "9d37e4d66cbd4d5b8a6df2045712856e"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-cgwx"
server_key: "7eb8ee68662e5a2ecba80415c872c450"
server_key_iv: "911cb41ebcc3765cb1e1549f2dd9df97"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.036563 18103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "9d37e4d66cbd4d5b8a6df2045712856e"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-cgwx"
server_key: "7eb8ee68662e5a2ecba80415c872c450"
server_key_iv: "911cb41ebcc3765cb1e1549f2dd9df97"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.040277 18103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:57.042586 18118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.043829 18103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:48:57.043972 18103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "9d37e4d66cbd4d5b8a6df2045712856e"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-cgwx"
server_key: "7eb8ee68662e5a2ecba80415c872c450"
server_key_iv: "911cb41ebcc3765cb1e1549f2dd9df97"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.044082 18103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:57.065864 18103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:57.069219 18103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:57.069437 18103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:57.070083 18103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:57.071017 18103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:57.071082 18103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.071149 18103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:57.071199 18103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.080545 18103 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:45709
I20260619 16:48:57.080566 18231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:45709 every 8 connection(s)
I20260619 16:48:57.081441 18103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:57.087070 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 18103
I20260619 16:48:57.087201 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:48:57.087529 4006 external_mini_cluster.cc:1428] Setting key 5492c4424c047004e1822e3fe258ee7a
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:57.096375 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.083316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:39741 (local address 127.3.233.190:34023)
0619 16:48:57.083484 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.083488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.084779 (+ 1291us) server_negotiation.cc:407] Connection header received
0619 16:48:57.085786 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.085789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.085837 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.085915 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:57.087284 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.088025 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.089650 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.089955 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.090040 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.092769 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:57.092772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:57.092774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:57.092798 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:57.094373 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.094875 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.094878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.094879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.095073 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.095430 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.095432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.095433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.095613 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:57.095696 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.096158 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.096258 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:48:57.097321 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.083682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:39741)
0619 16:48:57.084609 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.084657 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.085542 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.086087 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.086094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.086510 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:57.087119 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.087136 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.088229 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.088234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.089496 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.089508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.090099 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.090102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.090304 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.091038 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:57.091055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:57.092613 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:57.094480 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.094485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.094498 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.094779 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.095190 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.095192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.095194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.095334 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.095704 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:57.095709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:57.095937 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.096102 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.096373 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":548,"threads_started":1}
I20260619 16:48:57.098470 18232 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34023
I20260619 16:48:57.098692 18232 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:57.099242 18232 heartbeater.cc:511] Master 127.3.233.190:34023 requested a full tablet report, sending...
I20260619 16:48:57.100256 17773 ts_manager.cc:194] Registered new tserver with Master: 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709)
I20260619 16:48:57.100983 17773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:39741
I20260619 16:48:57.101806 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:48:57.111831 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.104351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.3.233.190:34023)
0619 16:48:57.104480 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.104484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.104691 (+ 207us) server_negotiation.cc:407] Connection header received
0619 16:48:57.104773 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.104775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.104817 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.104885 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:57.105629 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.106215 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.107258 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.107483 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.107575 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.108922 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:57.108925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:57.108927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:57.108949 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:57.110308 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.110690 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.110693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.110694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.110762 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.111045 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.111048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.111049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.111274 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:57.111377 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.111620 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.111711 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:48:57.114710 17773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:48:57.117686 17773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:48:57.132352 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.127067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.3.233.129:44361)
0619 16:48:57.127879 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.127895 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.127916 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:48:57.127988 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.127993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.128202 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.128357 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:57.128669 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.129632 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.131092 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.131685 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.131917 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.132063 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.132116 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.132213 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":88,"threads_started":1}
I20260619 16:48:57.132347 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.126876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44361 (local address 127.0.0.1:39028)
0619 16:48:57.127521 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.127533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.127628 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.128348 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.128354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.128374 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:57.128547 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.128552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.129758 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.129778 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.130979 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.130990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.131836 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.131839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.131938 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.131976 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.132077 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.132190 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":92,"spinlock_wait_cycles":8320,"thread_start_us":115,"threads_started":1}
I20260619 16:48:57.132594 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.126418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:45709 (local address 127.0.0.1:54288)
0619 16:48:57.127251 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.127266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.127353 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.128391 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.128394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.128406 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:57.128543 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.128550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.129956 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.129959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.131086 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.131095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.132103 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.132107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.132191 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.132212 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.132278 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.132346 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":47,"thread_start_us":80,"threads_started":1}
I20260619 16:48:57.133071 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.126164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34521 (local address 127.0.0.1:54572)
0619 16:48:57.126488 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.126557 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.126676 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.127574 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.127579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.127620 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:57.127767 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.127776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.129289 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.129292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.130654 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.130665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.131805 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.131812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.132790 (+ 978us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.132812 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.132882 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.132935 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":6,"spinlock_wait_cycles":6400,"thread_start_us":103,"threads_started":1}
I20260619 16:48:57.133195 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.126301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.3.233.130:34521)
0619 16:48:57.127065 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.127081 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.127108 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:48:57.127185 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.127190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.127404 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.127576 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:57.127903 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.129110 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.130782 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.131659 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.131773 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.131920 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.132940 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.133064 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":110,"threads_started":1}
I20260619 16:48:57.135583 18032 tablet_service.cc:1467] Processing CreateTablet for tablet f7b83ae3a0f74f049a82234baccb50d7 (DEFAULT_TABLE table=test-table [id=f9d73e8682bb4a9b8b7e0ce3721c63de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:57.135564 17898 tablet_service.cc:1467] Processing CreateTablet for tablet f7b83ae3a0f74f049a82234baccb50d7 (DEFAULT_TABLE table=test-table [id=f9d73e8682bb4a9b8b7e0ce3721c63de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:57.136205 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.126961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.3.233.131:45709)
0619 16:48:57.127910 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.127922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.127948 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:48:57.128008 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.128011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.128235 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.128695 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:57.128833 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.129806 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.131203 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.131986 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.132102 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.132224 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.132311 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.132398 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":116,"threads_started":1}
I20260619 16:48:57.136601 17898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b83ae3a0f74f049a82234baccb50d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:57.136608 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b83ae3a0f74f049a82234baccb50d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:57.138743 18166 tablet_service.cc:1467] Processing CreateTablet for tablet f7b83ae3a0f74f049a82234baccb50d7 (DEFAULT_TABLE table=test-table [id=f9d73e8682bb4a9b8b7e0ce3721c63de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:57.139786 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b83ae3a0f74f049a82234baccb50d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:57.142866 18249 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Bootstrap starting.
I20260619 16:48:57.144331 18250 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap starting.
I20260619 16:48:57.145468 18249 tablet_bootstrap.cc:654] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:57.146219 18249 log.cc:824] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:57.146517 18250 tablet_bootstrap.cc:654] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:57.146910 18252 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Bootstrap starting.
I20260619 16:48:57.147873 18250 log.cc:824] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:57.148913 18249 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: No bootstrap required, opened a new log
I20260619 16:48:57.149097 18249 ts_tablet_manager.cc:1397] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:48:57.149629 18252 tablet_bootstrap.cc:654] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:57.149793 18250 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: No bootstrap required, opened a new log
I20260619 16:48:57.149942 18250 ts_tablet_manager.cc:1397] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:48:57.150808 18252 log.cc:824] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:57.152742 18252 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: No bootstrap required, opened a new log
I20260619 16:48:57.152921 18252 ts_tablet_manager.cc:1397] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 16:48:57.153108 18250 raft_consensus.cc:348] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.153345 18250 raft_consensus.cc:374] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:57.153398 18250 raft_consensus.cc:729] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6002a1dc884bc7a35ce64f237862c3, State: Initialized, Role: FOLLOWER
I20260619 16:48:57.153317 18249 raft_consensus.cc:348] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.153582 18249 raft_consensus.cc:374] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:57.153641 18249 raft_consensus.cc:729] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3f18f3c6ed42ff89831f1fb15b7c73, State: Initialized, Role: FOLLOWER
I20260619 16:48:57.153916 18250 consensus_queue.cc:260] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.154060 18249 consensus_queue.cc:260] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.154879 18250 ts_tablet_manager.cc:1428] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:48:57.155012 18249 ts_tablet_manager.cc:1428] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:48:57.155251 17964 heartbeater.cc:503] Master 127.3.233.190:34023 was elected leader, sending a full tablet report...
I20260619 16:48:57.155360 18098 heartbeater.cc:503] Master 127.3.233.190:34023 was elected leader, sending a full tablet report...
I20260619 16:48:57.157651 18252 raft_consensus.cc:348] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.157948 18252 raft_consensus.cc:374] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:57.158001 18252 raft_consensus.cc:729] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d37e4d66cbd4d5b8a6df2045712856e, State: Initialized, Role: FOLLOWER
I20260619 16:48:57.158946 18252 consensus_queue.cc:260] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.159636 18232 heartbeater.cc:503] Master 127.3.233.190:34023 was elected leader, sending a full tablet report...
I20260619 16:48:57.159940 18252 ts_tablet_manager.cc:1428] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260619 16:48:57.194269 18257 raft_consensus.cc:482] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:48:57.194522 18257 raft_consensus.cc:504] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.195678 18257 leader_election.cc:288] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361)
I20260619 16:48:57.200461 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.196366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:52673 (local address 127.3.233.129:44361)
0619 16:48:57.196511 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.196516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.196677 (+ 161us) server_negotiation.cc:407] Connection header received
0619 16:48:57.196815 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.196818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.196866 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.196945 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:57.197194 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.197839 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.199197 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.199811 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.199923 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.199963 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.200249 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.200324 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:48:57.200461 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.196031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:57633 (local address 127.3.233.130:34521)
0619 16:48:57.196198 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.196202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.196218 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:48:57.196320 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.196323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.196371 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.196467 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:57.196771 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.197532 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.199061 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.199685 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.199816 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.199891 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.200250 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.200324 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:48:57.200532 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.195940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34521 (local address 127.3.233.131:57633)
0619 16:48:57.196097 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.196115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.196223 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.196487 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.196492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.196517 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:57.196650 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.196660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.197651 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.197654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.198913 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.198958 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.199831 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.199834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.200062 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.200126 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.200225 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.200327 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":34}
I20260619 16:48:57.200548 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.196218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44361 (local address 127.3.233.131:52673)
0619 16:48:57.196598 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.196615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.196716 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.196982 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.196984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.197000 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:57.197089 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.197094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.197936 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.197941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.199084 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.199091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.199923 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.199927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.200032 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.200142 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.200225 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.200327 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":53,"thread_start_us":136,"threads_started":1}
I20260619 16:48:57.201339 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b83ae3a0f74f049a82234baccb50d7" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6002a1dc884bc7a35ce64f237862c3" is_pre_election: true
I20260619 16:48:57.201361 18051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b83ae3a0f74f049a82234baccb50d7" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" is_pre_election: true
I20260619 16:48:57.201704 17918 raft_consensus.cc:2454] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 0.
I20260619 16:48:57.201709 18051 raft_consensus.cc:2454] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 0.
I20260619 16:48:57.202306 18122 leader_election.cc:302] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6002a1dc884bc7a35ce64f237862c3, 9d37e4d66cbd4d5b8a6df2045712856e; no voters:
I20260619 16:48:57.202648 18257 raft_consensus.cc:2781] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:48:57.202762 18257 raft_consensus.cc:482] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:48:57.202826 18257 raft_consensus.cc:3037] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:57.204221 18257 raft_consensus.cc:504] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.204632 18257 leader_election.cc:288] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 election: Requested vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361)
I20260619 16:48:57.205174 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b83ae3a0f74f049a82234baccb50d7" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6002a1dc884bc7a35ce64f237862c3"
I20260619 16:48:57.205222 18051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b83ae3a0f74f049a82234baccb50d7" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73"
I20260619 16:48:57.205294 17918 raft_consensus.cc:3037] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:57.205411 18051 raft_consensus.cc:3037] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:57.206535 17918 raft_consensus.cc:2454] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 1.
I20260619 16:48:57.206563 18051 raft_consensus.cc:2454] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 1.
I20260619 16:48:57.206851 18122 leader_election.cc:302] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6002a1dc884bc7a35ce64f237862c3, 9d37e4d66cbd4d5b8a6df2045712856e; no voters:
I20260619 16:48:57.207100 18257 raft_consensus.cc:2781] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:57.207341 18257 raft_consensus.cc:686] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 LEADER]: Becoming Leader. State: Replica: 9d37e4d66cbd4d5b8a6df2045712856e, State: Running, Role: LEADER
I20260619 16:48:57.207696 18257 consensus_queue.cc:237] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.211112 17772 catalog_manager.cc:5495] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131). New cstate: current_term: 1 leader_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } health_report { overall_health: HEALTHY } } }
I20260619 16:48:57.231768 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.228037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.3.233.131:45709)
0619 16:48:57.228186 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.228191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.228205 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:48:57.228371 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.228381 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.228452 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.228568 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:57.228789 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.229304 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.230213 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.230436 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.230545 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.230857 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:57.231355 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:57.231489 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.231530 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.231593 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:48:57.238158 18051 raft_consensus.cc:1261] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Refusing update from remote peer 9d37e4d66cbd4d5b8a6df2045712856e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:57.238158 17918 raft_consensus.cc:1261] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Refusing update from remote peer 9d37e4d66cbd4d5b8a6df2045712856e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:57.238895 18257 consensus_queue.cc:1035] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:57.239122 18259 consensus_queue.cc:1035] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:57.247133 18263 mvcc.cc:205] Tried to move back new op lower bound from 7298612171720298496 to 7298612171605635072. Current Snapshot: MvccSnapshot[applied={T|T < 7298612171720298496}]
I20260619 16:48:57.250150 18264 mvcc.cc:205] Tried to move back new op lower bound from 7298612171720298496 to 7298612171605635072. Current Snapshot: MvccSnapshot[applied={T|T < 7298612171720298496}]
I20260619 16:48:57.254909 17772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:57.255148 17772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:57.257673 17772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Sending DeleteTablet for 3 replicas of tablet f7b83ae3a0f74f049a82234baccb50d7
I20260619 16:48:57.258513 18032 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b83ae3a0f74f049a82234baccb50d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:57 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:54572
I20260619 16:48:57.258504 17898 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b83ae3a0f74f049a82234baccb50d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:57 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:39028
I20260619 16:48:57.258746 18166 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b83ae3a0f74f049a82234baccb50d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:57 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:54288
I20260619 16:48:57.259063 18273 tablet_replica.cc:330] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: stopping tablet replica
I20260619 16:48:57.259351 18274 tablet_replica.cc:330] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: stopping tablet replica
I20260619 16:48:57.259366 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17835
I20260619 16:48:57.259395 18273 raft_consensus.cc:2229] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:57.260373 18274 raft_consensus.cc:2229] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:48:57.260471 18272 tablet_replica.cc:330] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: stopping tablet replica
I20260619 16:48:57.260742 18272 raft_consensus.cc:2229] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:57.260787 18274 raft_consensus.cc:2258] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:57.261046 18272 raft_consensus.cc:2258] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:57.262182 18274 ts_tablet_manager.cc:1905] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:57.262472 18272 ts_tablet_manager.cc:1905] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:57.264671 18274 ts_tablet_manager.cc:1918] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:57.264801 18274 log.cc:1196] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/f7b83ae3a0f74f049a82234baccb50d7
I20260619 16:48:57.265148 18272 ts_tablet_manager.cc:1918] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:57.265182 18274 ts_tablet_manager.cc:1939] T f7b83ae3a0f74f049a82234baccb50d7 P 9d37e4d66cbd4d5b8a6df2045712856e: Deleting consensus metadata
I20260619 16:48:57.265244 18272 log.cc:1196] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/f7b83ae3a0f74f049a82234baccb50d7
I20260619 16:48:57.265542 18272 ts_tablet_manager.cc:1939] T f7b83ae3a0f74f049a82234baccb50d7 P ad3f18f3c6ed42ff89831f1fb15b7c73: Deleting consensus metadata
I20260619 16:48:57.266474 17757 catalog_manager.cc:4841] TS 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709): tablet f7b83ae3a0f74f049a82234baccb50d7 (table test-table [id=f9d73e8682bb4a9b8b7e0ce3721c63de]) successfully deleted
I20260619 16:48:57.266737 17760 catalog_manager.cc:4841] TS ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521): tablet f7b83ae3a0f74f049a82234baccb50d7 (table test-table [id=f9d73e8682bb4a9b8b7e0ce3721c63de]) successfully deleted
W20260619 16:48:57.267220 17760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:44361 (error 108)
I20260619 16:48:57.267506 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17969
I20260619 16:48:57.271920 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.271542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44361 (local address 127.0.0.1:39032)
0619 16:48:57.271721 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.271808 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 16:48:57.272200 17760 catalog_manager.cc:4584] TS 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7b83ae3a0f74f049a82234baccb50d7: Network error: Client connection negotiation failed: client connection to 127.3.233.129:44361: connect: Connection refused (error 111)
I20260619 16:48:57.273205 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 18103
I20260619 16:48:57.279338 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 17741
I20260619 16:48:57.285724 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34023
--webserver_interface=127.3.233.190
--webserver_port=44705
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:57.396047 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:57.396322 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:57.396395 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:57.400401 18276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:57.400455 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:57.400475 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:57.400538 18276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:57.400573 18276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:57.406487 18276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=44705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:57.407670 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:57.408712 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:57.422230 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.422188 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.422235 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:57.422487 18276 server_base.cc:1034] running on GCE node
I20260619 16:48:57.423220 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:57.424504 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:57.425750 18276 hybrid_clock.cc:648] HybridClock initialized: now 1781887737425711 us; error 56 us; skew 500 ppm
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:57.428802 18276 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:48:57.429836 18276 webserver.cc:460] Webserver started at http://127.3.233.190:44705/ using document root <none> and password file <none>
I20260619 16:48:57.430454 18276 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:57.430570 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:57.434684 18276 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:57.436952 18292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.438287 18276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:48:57.438442 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "f73aff31c41749f5a1740f1ba1a79287"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "99999d84e7e59bc12d52cf6fc455a594"
server_key_iv: "9022900364f79811affcc8b399907587"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.439105 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:57.457957 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:57.461369 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:57.461616 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:57.470091 18276 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34023
I20260619 16:48:57.470379 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34023 every 8 connection(s)
I20260619 16:48:57.471431 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:57.472411 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 18276
I20260619 16:48:57.472884 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:44361
--local_ip_for_outbound_sockets=127.3.233.129
--tserver_master_addrs=127.3.233.190:34023
--webserver_port=42785
--webserver_interface=127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:57.476141 18345 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:48:57.477803 18345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Bootstrap starting.
I20260619 16:48:57.495805 18345 log.cc:824] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:57.504779 18345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:48:57.505249 18345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Bootstrap complete.
I20260619 16:48:57.509461 18345 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:57.509903 18345 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73aff31c41749f5a1740f1ba1a79287, State: Initialized, Role: FOLLOWER
I20260619 16:48:57.510407 18345 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:57.510583 18345 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:57.510674 18345 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:57.510800 18345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:48:57.512770 18345 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:57.513113 18345 leader_election.cc:302] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73aff31c41749f5a1740f1ba1a79287; no voters:
I20260619 16:48:57.513433 18345 leader_election.cc:288] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:48:57.513584 18349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:48:57.513839 18349 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 LEADER]: Becoming Leader. State: Replica: f73aff31c41749f5a1740f1ba1a79287, State: Running, Role: LEADER
I20260619 16:48:57.514145 18349 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:57.514676 18345 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:57.515591 18350 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f73aff31c41749f5a1740f1ba1a79287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } } }
I20260619 16:48:57.515745 18350 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:57.516120 18351 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73aff31c41749f5a1740f1ba1a79287. Latest consensus state: current_term: 2 leader_uuid: "f73aff31c41749f5a1740f1ba1a79287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } } }
I20260619 16:48:57.516263 18351 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:57.516575 18358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:57.520506 18358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:57.521032 18358 catalog_manager.cc:1259] Loaded cluster ID: 419a48323e4d427ab5ff31327bb1c857
I20260619 16:48:57.521111 18358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:48:57.522176 18358 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 16:48:57.525063 18358 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 16:48:57.529309 18358 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:48:57.529994 18358 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Loaded TSK: 0
I20260619 16:48:57.530989 18358 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:48:57.601019 18347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:57.601305 18347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:57.601351 18347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:57.605726 18347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:57.605799 18347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:57.605860 18347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:48:57.611312 18347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:44361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=42785
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:57.612370 18347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:57.613435 18347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:57.623598 18373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:57.623824 18347 server_base.cc:1034] running on GCE node
W20260619 16:48:57.623598 18372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.623611 18375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:57.624590 18347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:57.625442 18347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:57.626624 18347 hybrid_clock.cc:648] HybridClock initialized: now 1781887737626596 us; error 60 us; skew 500 ppm
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:57.629902 18347 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:48:57.630837 18347 webserver.cc:460] Webserver started at http://127.3.233.129:42785/ using document root <none> and password file <none>
I20260619 16:48:57.631435 18347 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:57.631546 18347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:57.635628 18347 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:48:57.637917 18382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.639406 18347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:48:57.639530 18347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "0b6002a1dc884bc7a35ce64f237862c3"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "ba38c04949cc6f508c77b3d9cab79432"
server_key_iv: "9b86760a2f4b91da068631c5f1c2c93d"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.640074 18347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:57.651866 18347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:57.655090 18347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:57.655284 18347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:57.655859 18347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:57.656926 18389 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:48:57.662315 18347 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:48:57.662412 18347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 16:48:57.662499 18347 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:48:57.664322 18347 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:48:57.664399 18347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 16:48:57.664690 18389 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap starting.
I20260619 16:48:57.675127 18347 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:44361
I20260619 16:48:57.675309 18496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:44361 every 8 connection(s)
I20260619 16:48:57.676165 18347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:48:57.680058 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 18347
I20260619 16:48:57.680488 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:34521
--local_ip_for_outbound_sockets=127.3.233.130
--tserver_master_addrs=127.3.233.190:34023
--webserver_port=42897
--webserver_interface=127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:57.684791 18389 log.cc:824] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:57.693091 18389 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:48:57.693552 18389 tablet_bootstrap.cc:492] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap complete.
I20260619 16:48:57.695039 18389 ts_tablet_manager.cc:1397] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20260619 16:48:57.696352 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.678248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:60521 (local address 127.3.233.190:34023)
0619 16:48:57.678679 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.678700 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.678964 (+ 264us) server_negotiation.cc:407] Connection header received
0619 16:48:57.680133 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.680148 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.680546 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.680923 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:57.683078 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.684750 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.686199 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.687058 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.687243 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.690392 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:57.690399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:57.690415 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:57.690447 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:57.692653 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.693290 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.693299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.693308 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.693470 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.693858 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.693864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.693869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.694365 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:57.694500 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.695167 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.695475 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
I20260619 16:48:57.697396 18389 raft_consensus.cc:348] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.697783 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.678162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:60521)
0619 16:48:57.678765 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.678817 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.679645 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.681606 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.681615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.682101 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:57.682857 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.682878 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.684932 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.684935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.686061 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.686072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.687225 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.687229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.687431 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.688152 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:57.688173 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:57.690168 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:57.692787 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.692794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.692807 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.693140 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.693579 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.693582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.693584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.693751 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.694481 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:57.694487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:57.694776 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.696608 (+ 1832us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.696939 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":2,"thread_start_us":134,"threads_started":1}
I20260619 16:48:57.697827 18389 raft_consensus.cc:729] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6002a1dc884bc7a35ce64f237862c3, State: Initialized, Role: FOLLOWER
I20260619 16:48:57.698297 18389 consensus_queue.cc:260] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } }
I20260619 16:48:57.699258 18389 ts_tablet_manager.cc:1428] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:48:57.700205 18497 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34023
I20260619 16:48:57.700454 18497 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:57.700947 18497 heartbeater.cc:511] Master 127.3.233.190:34023 requested a full tablet report, sending...
I20260619 16:48:57.703606 18309 ts_manager.cc:194] Registered new tserver with Master: 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361)
I20260619 16:48:57.707788 18309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:60521
I20260619 16:48:57.710676 18497 heartbeater.cc:503] Master 127.3.233.190:34023 was elected leader, sending a full tablet report...
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.129@KRBTEST.COM
I20260619 16:48:57.718338 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.707296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.3.233.129:44361)
0619 16:48:57.707734 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.707746 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.708197 (+ 451us) server_negotiation.cc:407] Connection header received
0619 16:48:57.708412 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.708422 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.708635 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.708802 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:57.709493 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.710204 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.711252 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.711744 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.711862 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.714563 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:57.714572 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:57.714582 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:57.714605 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:57.716427 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.716969 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.716973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.716975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.717056 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.717356 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.717359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.717360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.717599 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:57.717701 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.718064 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.718211 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":114,"threads_started":1}
I20260619 16:48:57.718362 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.707274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44361 (local address 127.0.0.1:39042)
0619 16:48:57.708051 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.708087 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.708253 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.708802 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.708808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.709231 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:57.709373 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.709381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.710318 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.710323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.711158 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.711170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.711877 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.711880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.711996 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.712699 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:57.712714 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:57.714391 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:57.716531 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.716541 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.716547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.716852 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.717159 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.717162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.717164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.717264 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.717706 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:57.717712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:57.717842 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.717988 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.718193 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":115,"threads_started":1}
I20260619 16:48:57.719456 18431 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b83ae3a0f74f049a82234baccb50d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:57 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:39042
I20260619 16:48:57.720533 18506 tablet_replica.cc:330] stopping tablet replica
I20260619 16:48:57.720844 18506 raft_consensus.cc:2229] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:57.721093 18506 raft_consensus.cc:2258] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:57.722090 18506 ts_tablet_manager.cc:1905] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:57.725570 18506 ts_tablet_manager.cc:1918] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:57.725684 18506 log.cc:1196] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/f7b83ae3a0f74f049a82234baccb50d7
I20260619 16:48:57.725962 18506 ts_tablet_manager.cc:1939] T f7b83ae3a0f74f049a82234baccb50d7 P 0b6002a1dc884bc7a35ce64f237862c3: Deleting consensus metadata
I20260619 16:48:57.727674 18296 catalog_manager.cc:4841] TS 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361): tablet f7b83ae3a0f74f049a82234baccb50d7 (table test-table [id=f9d73e8682bb4a9b8b7e0ce3721c63de]) successfully deleted
W20260619 16:48:57.803553 18501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:57.803820 18501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:57.803864 18501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:57.807961 18501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:57.808020 18501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:57.808079 18501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:48:57.813258 18501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:34521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=42897
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:57.814313 18501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:57.815485 18501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:57.825706 18511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.825706 18514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:57.825729 18512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:57.826051 18501 server_base.cc:1034] running on GCE node
I20260619 16:48:57.826686 18501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:57.827468 18501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:57.828639 18501 hybrid_clock.cc:648] HybridClock initialized: now 1781887737828623 us; error 32 us; skew 500 ppm
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:57.831626 18501 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:48:57.832615 18501 webserver.cc:460] Webserver started at http://127.3.233.130:42897/ using document root <none> and password file <none>
I20260619 16:48:57.833173 18501 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:57.833250 18501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:57.837422 18501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:57.839633 18521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.840922 18501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:48:57.841069 18501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "7935fdd500bffd1089bdaf0743dea55a"
server_key_iv: "6ee614b85bac2b5ac46519ae1e0079fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:57.841655 18501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:57.853533 18501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:57.856791 18501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:57.857019 18501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:57.857656 18501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:57.858539 18501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:57.858618 18501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.858690 18501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:57.858749 18501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:57.868845 18501 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:34521
I20260619 16:48:57.868862 18634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:34521 every 8 connection(s)
I20260619 16:48:57.869963 18501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:57.877485 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 18501
I20260619 16:48:57.877954 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:45709
--local_ip_for_outbound_sockets=127.3.233.131
--tserver_master_addrs=127.3.233.190:34023
--webserver_port=46047
--webserver_interface=127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:48:57 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887737, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:57.885409 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.872029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:44543 (local address 127.3.233.190:34023)
0619 16:48:57.872197 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:57.872202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:57.873292 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 16:48:57.874235 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:57.874238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:57.874295 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:57.874443 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:57.875707 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.876242 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.877481 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.877813 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.877911 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.881195 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:57.881200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:57.881202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:57.881227 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:57.882823 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.883579 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.883582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.883583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.883662 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:57.884096 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:57.884099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:57.884100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:57.884278 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:57.884366 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:57.885161 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 16:48:57.885276 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:48:57.886650 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:57.872446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:44543)
0619 16:48:57.873117 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:57.873167 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:57.874014 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:57.874618 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:57.874631 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:57.875076 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:57.875551 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.875564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.876369 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.876373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.877343 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:57.877357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:57.877975 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:57.877980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:57.878217 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:57.878863 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:57.878888 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:57.881007 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:57.882981 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.882990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.883010 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.883415 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.883770 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:57.883775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:57.883778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:57.883980 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:57.884389 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:57.884398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:57.884759 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 16:48:57.885048 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 16:48:57.885446 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":123,"threads_started":1}
I20260619 16:48:57.887954 18635 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34023
I20260619 16:48:57.888249 18635 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:57.888726 18635 heartbeater.cc:511] Master 127.3.233.190:34023 requested a full tablet report, sending...
I20260619 16:48:57.889751 18309 ts_manager.cc:194] Registered new tserver with Master: ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521)
I20260619 16:48:57.890483 18309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:44543
W20260619 16:48:57.992825 18638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:57.993113 18638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:57.993160 18638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:57.997433 18638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:57.997496 18638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:57.997556 18638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:48:58.002775 18638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:45709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=46047
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:58.003968 18638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:58.005045 18638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:58.014654 18644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.014732 18643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.014813 18638 server_base.cc:1034] running on GCE node
W20260619 16:48:58.014671 18646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.015293 18638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:58.016023 18638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:58.017200 18638 hybrid_clock.cc:648] HybridClock initialized: now 1781887738017169 us; error 44 us; skew 500 ppm
Jun 19 16:48:58 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:58.020025 18638 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:48:58.020881 18638 webserver.cc:460] Webserver started at http://127.3.233.131:46047/ using document root <none> and password file <none>
I20260619 16:48:58.021636 18638 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:58.021737 18638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:58.025883 18638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:58.028040 18653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.029294 18638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:48:58.029438 18638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "9d37e4d66cbd4d5b8a6df2045712856e"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-cgwx"
server_key: "7eb8ee68662e5a2ecba80415c872c450"
server_key_iv: "911cb41ebcc3765cb1e1549f2dd9df97"
server_key_version: "encryptionkey@0"
I20260619 16:48:58.030007 18638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:58.045413 18638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:58.048559 18638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:58.048766 18638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:58.049358 18638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:58.050160 18638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:58.050212 18638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.050280 18638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:58.050319 18638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.059732 18638 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:45709
I20260619 16:48:58.059756 18766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:45709 every 8 connection(s)
I20260619 16:48:58.060698 18638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:58.065508 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 18638
Jun 19 16:48:58 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:48:58.074753 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.066226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.3.233.190:34023)
0619 16:48:58.066476 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.066481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.066700 (+ 219us) server_negotiation.cc:407] Connection header received
0619 16:48:58.066740 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.066744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.066789 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.066885 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.067776 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.068288 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.069289 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.069488 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.069560 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.071063 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.071067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.071069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.071093 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.073073 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.073510 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.073517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.073521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.073607 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.073886 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.073891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.073893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.074078 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.074300 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.074446 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.074573 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260619 16:48:58.075639 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.062577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:42875 (local address 127.3.233.190:34023)
0619 16:48:58.062732 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.062736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.063856 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 16:48:58.064819 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.064822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.064885 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.065018 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:48:58.066641 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.067345 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.068753 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.068954 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.069028 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.072134 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:48:58.072138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:48:58.072139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:48:58.072169 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:48:58.073512 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.074034 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.074038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.074040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.074116 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:48:58.074452 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:48:58.074457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:48:58.074458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:48:58.074616 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:48:58.074734 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.075385 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.075487 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:48:58.076709 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.062996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:42875)
0619 16:48:58.063689 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.063733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:58.064592 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:58.065194 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:58.065223 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:58.065752 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:48:58.066406 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.066431 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.067475 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.067478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.068634 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.068646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.069101 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.069105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.069309 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.070011 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:48:58.070035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:48:58.071966 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:48:58.073631 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:58.073637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:58.073652 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:58.073930 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:58.074217 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:48:58.074220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:48:58.074222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:48:58.074357 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:48:58.074790 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:48:58.074796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:48:58.075131 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 16:48:58.075336 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 16:48:58.075647 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":108,"threads_started":1}
I20260619 16:48:58.077769 18767 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34023
I20260619 16:48:58.078048 18767 heartbeater.cc:464] Registering TS with master...
I20260619 16:48:58.078591 18767 heartbeater.cc:511] Master 127.3.233.190:34023 requested a full tablet report, sending...
I20260619 16:48:58.079690 18309 ts_manager.cc:194] Registered new tserver with Master: 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709)
I20260619 16:48:58.080310 18309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:42875
I20260619 16:48:58.088488 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:48:58.093057 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.089191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.3.233.190:34023)
0619 16:48:58.089329 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.089332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.089505 (+ 173us) server_negotiation.cc:407] Connection header received
0619 16:48:58.089649 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.089651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.089757 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.089845 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:58.090158 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.090649 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.091715 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.091928 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.092014 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.092300 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:58.092754 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:58.092845 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.092909 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.092952 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:48:58.093748 18309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:48:58.097267 18309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:48:58.106181 18431 tablet_service.cc:1467] Processing CreateTablet for tablet dee09f4e51c4490baae7475a22dd6302 (DEFAULT_TABLE table=test-table [id=ec0e4c9022814de0b189d791c338f9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:58.106848 18431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee09f4e51c4490baae7475a22dd6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:58.109762 18389 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap starting.
I20260619 16:48:58.110440 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.104748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34521 (local address 127.0.0.1:54586)
0619 16:48:58.104884 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.104903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:58.105008 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:58.106135 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:58.106138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:58.106158 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:58.106275 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.106282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.107750 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.107753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.108968 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.108978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.110007 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.110010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.110102 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.110157 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 16:48:58.110214 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 16:48:58.110301 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260619 16:48:58.110770 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.105062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:45709 (local address 127.0.0.1:54308)
0619 16:48:58.105716 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.105730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:58.105829 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:58.106740 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:58.106743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:58.106762 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:58.106846 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.106852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.108337 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.108343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.109128 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.109146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.110380 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.110384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.110470 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.110492 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:48:58.110563 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:48:58.110632 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":164,"threads_started":1}
I20260619 16:48:58.110782 18389 tablet_bootstrap.cc:654] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:58.111251 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.105312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.3.233.131:45709)
0619 16:48:58.105651 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.105661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.105823 (+ 162us) server_negotiation.cc:407] Connection header received
0619 16:48:58.105941 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.105947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.106320 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.106546 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:58.106999 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.108206 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.109275 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.110222 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.110341 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.110723 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.110785 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.111103 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20260619 16:48:58.111374 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.104956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.3.233.130:34521)
0619 16:48:58.105376 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.105387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.105409 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:48:58.105509 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.105514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.105768 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.106007 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:58.106394 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.107597 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.109114 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.109869 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.110859 (+ 990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.111052 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.111110 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.111223 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":217,"threads_started":1}
I20260619 16:48:58.112397 18389 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: No bootstrap required, opened a new log
I20260619 16:48:58.112471 18389 ts_tablet_manager.cc:1397] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:58.112882 18389 raft_consensus.cc:348] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.112977 18389 raft_consensus.cc:374] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:58.113003 18389 raft_consensus.cc:729] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6002a1dc884bc7a35ce64f237862c3, State: Initialized, Role: FOLLOWER
I20260619 16:48:58.113160 18389 consensus_queue.cc:260] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.114497 18701 tablet_service.cc:1467] Processing CreateTablet for tablet dee09f4e51c4490baae7475a22dd6302 (DEFAULT_TABLE table=test-table [id=ec0e4c9022814de0b189d791c338f9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:58.115595 18701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee09f4e51c4490baae7475a22dd6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:58.115584 18569 tablet_service.cc:1467] Processing CreateTablet for tablet dee09f4e51c4490baae7475a22dd6302 (DEFAULT_TABLE table=test-table [id=ec0e4c9022814de0b189d791c338f9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:48:58.116525 18569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee09f4e51c4490baae7475a22dd6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:48:58.122525 18780 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Bootstrap starting.
I20260619 16:48:58.123416 18389 ts_tablet_manager.cc:1428] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20260619 16:48:58.123405 18781 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Bootstrap starting.
I20260619 16:48:58.125811 18781 tablet_bootstrap.cc:654] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:58.125813 18780 tablet_bootstrap.cc:654] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Neither blocks nor log segments found. Creating new log.
I20260619 16:48:58.126549 18781 log.cc:824] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:58.126549 18780 log.cc:824] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:58.128420 18780 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: No bootstrap required, opened a new log
I20260619 16:48:58.128443 18781 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: No bootstrap required, opened a new log
I20260619 16:48:58.128614 18780 ts_tablet_manager.cc:1397] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 16:48:58.128633 18781 ts_tablet_manager.cc:1397] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:48:58.131734 18780 raft_consensus.cc:348] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.131734 18781 raft_consensus.cc:348] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.131956 18780 raft_consensus.cc:374] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:58.131984 18781 raft_consensus.cc:374] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:48:58.132018 18780 raft_consensus.cc:729] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3f18f3c6ed42ff89831f1fb15b7c73, State: Initialized, Role: FOLLOWER
I20260619 16:48:58.132050 18781 raft_consensus.cc:729] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d37e4d66cbd4d5b8a6df2045712856e, State: Initialized, Role: FOLLOWER
I20260619 16:48:58.132486 18780 consensus_queue.cc:260] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.132488 18781 consensus_queue.cc:260] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.133126 18767 heartbeater.cc:503] Master 127.3.233.190:34023 was elected leader, sending a full tablet report...
I20260619 16:48:58.133446 18780 ts_tablet_manager.cc:1428] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 16:48:58.133473 18781 ts_tablet_manager.cc:1428] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:48:58.133803 18635 heartbeater.cc:503] Master 127.3.233.190:34023 was elected leader, sending a full tablet report...
I20260619 16:48:58.157045 18784 raft_consensus.cc:482] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:48:58.157320 18784 raft_consensus.cc:504] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.158432 18784 leader_election.cc:288] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361)
I20260619 16:48:58.163605 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.159036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:60281 (local address 127.3.233.130:34521)
0619 16:48:58.159202 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.159207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.159224 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:48:58.159286 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.159289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.159343 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.159433 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:58.159758 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.160549 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.162137 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.162783 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.162940 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.162996 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.163403 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.163479 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:48:58.163606 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.159284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:49569 (local address 127.3.233.129:44361)
0619 16:48:58.159424 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.159429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.159578 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:48:58.159689 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.159692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.159761 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.159866 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:48:58.160126 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.160816 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.162165 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.162817 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.162943 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.163090 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.163403 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.163479 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:48:58.163610 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.158814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34521 (local address 127.3.233.131:60281)
0619 16:48:58.158980 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.158999 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:58.159087 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:58.159454 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:58.159458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:58.159478 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:58.159610 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.159620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.160670 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.160673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.161964 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.161975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.162957 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.162961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.163252 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.163287 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:48:58.163360 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:48:58.163447 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":81}
I20260619 16:48:58.163626 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.159112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44361 (local address 127.3.233.131:49569)
0619 16:48:58.159464 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.159476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:58.159588 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:58.159890 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:48:58.159892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:48:58.159909 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:48:58.160006 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.160011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.160911 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.160915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.162049 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:48:58.162059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.162957 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.162961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:48:58.163224 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.163275 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 16:48:58.163360 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:48:58.163447 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":121,"thread_start_us":154,"threads_started":1}
I20260619 16:48:58.164316 18589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee09f4e51c4490baae7475a22dd6302" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" is_pre_election: true
I20260619 16:48:58.164283 18451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee09f4e51c4490baae7475a22dd6302" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6002a1dc884bc7a35ce64f237862c3" is_pre_election: true
I20260619 16:48:58.164613 18589 raft_consensus.cc:2454] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 0.
I20260619 16:48:58.164613 18451 raft_consensus.cc:2454] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 0.
I20260619 16:48:58.165134 18657 leader_election.cc:302] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d37e4d66cbd4d5b8a6df2045712856e, ad3f18f3c6ed42ff89831f1fb15b7c73; no voters:
I20260619 16:48:58.165437 18784 raft_consensus.cc:2781] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:48:58.165524 18784 raft_consensus.cc:482] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:48:58.165568 18784 raft_consensus.cc:3037] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:58.166587 18784 raft_consensus.cc:504] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.166973 18784 leader_election.cc:288] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 election: Requested vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361)
I20260619 16:48:58.167390 18589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee09f4e51c4490baae7475a22dd6302" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73"
I20260619 16:48:58.167403 18451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee09f4e51c4490baae7475a22dd6302" candidate_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6002a1dc884bc7a35ce64f237862c3"
I20260619 16:48:58.167548 18589 raft_consensus.cc:3037] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:58.167543 18451 raft_consensus.cc:3037] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:48:58.168809 18589 raft_consensus.cc:2454] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 1.
I20260619 16:48:58.168820 18451 raft_consensus.cc:2454] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d37e4d66cbd4d5b8a6df2045712856e in term 1.
I20260619 16:48:58.169239 18657 leader_election.cc:302] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d37e4d66cbd4d5b8a6df2045712856e, ad3f18f3c6ed42ff89831f1fb15b7c73; no voters:
I20260619 16:48:58.169436 18784 raft_consensus.cc:2781] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:48:58.169693 18784 raft_consensus.cc:686] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 LEADER]: Becoming Leader. State: Replica: 9d37e4d66cbd4d5b8a6df2045712856e, State: Running, Role: LEADER
I20260619 16:48:58.170064 18784 consensus_queue.cc:237] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.173449 18307 catalog_manager.cc:5495] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131). New cstate: current_term: 1 leader_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } health_report { overall_health: UNKNOWN } } }
W20260619 16:48:58.177392 18498 tablet.cc:2369] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:48:58.194654 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.190093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.3.233.131:45709)
0619 16:48:58.190339 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 16:48:58.190345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:48:58.190362 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:48:58.190456 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:48:58.190458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:48:58.190515 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:48:58.190639 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:48:58.190884 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.191416 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.192690 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:48:58.193043 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:48:58.193171 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:48:58.193524 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:48:58.194155 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:48:58.194284 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:48:58.194406 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 16:48:58.194473 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260619 16:48:58.200793 18451 raft_consensus.cc:1261] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Refusing update from remote peer 9d37e4d66cbd4d5b8a6df2045712856e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:58.200796 18589 raft_consensus.cc:1261] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Refusing update from remote peer 9d37e4d66cbd4d5b8a6df2045712856e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:48:58.201479 18784 consensus_queue.cc:1035] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:58.201743 18787 consensus_queue.cc:1035] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:48:58.208756 18791 mvcc.cc:205] Tried to move back new op lower bound from 7298612175663468544 to 7298612175547432960. Current Snapshot: MvccSnapshot[applied={T|T < 7298612175663468544}]
I20260619 16:48:58.216631 18307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:58.216809 18307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:48:58.219163 18307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Sending DeleteTablet for 3 replicas of tablet dee09f4e51c4490baae7475a22dd6302
I20260619 16:48:58.219889 18431 tablet_service.cc:1514] Processing DeleteTablet for tablet dee09f4e51c4490baae7475a22dd6302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:58 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:39042
I20260619 16:48:58.219868 18569 tablet_service.cc:1514] Processing DeleteTablet for tablet dee09f4e51c4490baae7475a22dd6302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:58 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:54586
I20260619 16:48:58.220093 18701 tablet_service.cc:1514] Processing DeleteTablet for tablet dee09f4e51c4490baae7475a22dd6302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:48:58 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:54308
I20260619 16:48:58.220446 18799 tablet_replica.cc:330] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: stopping tablet replica
I20260619 16:48:58.220520 18506 tablet_replica.cc:330] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: stopping tablet replica
I20260619 16:48:58.220584 18800 tablet_replica.cc:330] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: stopping tablet replica
I20260619 16:48:58.220664 18506 raft_consensus.cc:2229] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:58.220798 18506 raft_consensus.cc:2258] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:58.220788 18799 raft_consensus.cc:2229] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:48:58.220790 18800 raft_consensus.cc:2229] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:48:58.221138 18799 raft_consensus.cc:2258] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:58.221143 18800 raft_consensus.cc:2258] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:48:58.221403 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 18347
I20260619 16:48:58.224395 18799 ts_tablet_manager.cc:1905] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:48:58.226558 18799 ts_tablet_manager.cc:1918] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:58.226640 18799 log.cc:1196] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/dee09f4e51c4490baae7475a22dd6302
I20260619 16:48:58.226882 18799 ts_tablet_manager.cc:1939] T dee09f4e51c4490baae7475a22dd6302 P ad3f18f3c6ed42ff89831f1fb15b7c73: Deleting consensus metadata
I20260619 16:48:58.228332 18296 catalog_manager.cc:4841] TS ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521): tablet dee09f4e51c4490baae7475a22dd6302 (table test-table [id=ec0e4c9022814de0b189d791c338f9bf]) successfully deleted
I20260619 16:48:58.228559 18800 ts_tablet_manager.cc:1905] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:48:58.229705 18296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:44361 (error 108)
I20260619 16:48:58.229976 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 18501
I20260619 16:48:58.231323 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.231055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:44361 (local address 127.0.0.1:39044)
0619 16:48:58.231171 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.231244 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:44361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 16:48:58.231598 18800 ts_tablet_manager.cc:1918] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:48:58.231700 18800 log.cc:1196] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/dee09f4e51c4490baae7475a22dd6302
I20260619 16:48:58.232023 18800 ts_tablet_manager.cc:1939] T dee09f4e51c4490baae7475a22dd6302 P 9d37e4d66cbd4d5b8a6df2045712856e: Deleting consensus metadata
W20260619 16:48:58.231556 18296 catalog_manager.cc:4584] TS 0b6002a1dc884bc7a35ce64f237862c3 (127.3.233.129:44361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dee09f4e51c4490baae7475a22dd6302: Network error: Client connection negotiation failed: client connection to 127.3.233.129:44361: connect: Connection refused (error 111)
I20260619 16:48:58.233770 18293 catalog_manager.cc:4841] TS 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709): tablet dee09f4e51c4490baae7475a22dd6302 (table test-table [id=ec0e4c9022814de0b189d791c338f9bf]) successfully deleted
I20260619 16:48:58.237883 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 18638
I20260619 16:48:58.243402 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 18276
I20260619 16:48:58.250204 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34023
--webserver_interface=127.3.233.190
--webserver_port=44705
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:48:58.361765 18802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:58.362083 18802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:58.362151 18802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:58.366161 18802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:48:58.366259 18802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:58.366298 18802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:58.366334 18802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:48:58.366369 18802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:48:58.371554 18802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=44705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:58.372627 18802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:58.373665 18802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:58.382180 18808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.382239 18802 server_base.cc:1034] running on GCE node
W20260619 16:48:58.382160 18807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.382160 18810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.383111 18802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:58.384315 18802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:58.385522 18802 hybrid_clock.cc:648] HybridClock initialized: now 1781887738385511 us; error 33 us; skew 500 ppm
Jun 19 16:48:58 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:58.388382 18802 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:48:58.389354 18802 webserver.cc:460] Webserver started at http://127.3.233.190:44705/ using document root <none> and password file <none>
I20260619 16:48:58.389932 18802 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:58.390025 18802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:58.393966 18802 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:48:58.396116 18817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.397423 18802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:48:58.397552 18802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "f73aff31c41749f5a1740f1ba1a79287"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "99999d84e7e59bc12d52cf6fc455a594"
server_key_iv: "9022900364f79811affcc8b399907587"
server_key_version: "encryptionkey@0"
I20260619 16:48:58.398078 18802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:58.409616 18802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:58.412744 18802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:58.413023 18802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:58.419979 18802 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34023
I20260619 16:48:58.420300 18869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34023 every 8 connection(s)
I20260619 16:48:58.421072 18802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:48:58.425416 18870 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:48:58.426240 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 18802
I20260619 16:48:58.426736 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:44361
--local_ip_for_outbound_sockets=127.3.233.129
--tserver_master_addrs=127.3.233.190:34023
--webserver_port=42785
--webserver_interface=127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:58.427574 18870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Bootstrap starting.
I20260619 16:48:58.444947 18870 log.cc:824] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:58.456063 18870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:48:58.456442 18870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287: Bootstrap complete.
I20260619 16:48:58.460492 18870 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:58.460974 18870 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73aff31c41749f5a1740f1ba1a79287, State: Initialized, Role: FOLLOWER
I20260619 16:48:58.461511 18870 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:58.461656 18870 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:48:58.461747 18870 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:48:58.461848 18870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:48:58.464252 18870 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:58.464653 18870 leader_election.cc:302] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73aff31c41749f5a1740f1ba1a79287; no voters:
I20260619 16:48:58.465025 18870 leader_election.cc:288] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:48:58.465113 18874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:48:58.465319 18874 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [term 3 LEADER]: Becoming Leader. State: Replica: f73aff31c41749f5a1740f1ba1a79287, State: Running, Role: LEADER
I20260619 16:48:58.465626 18874 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } }
I20260619 16:48:58.466323 18870 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:48:58.466617 18874 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73aff31c41749f5a1740f1ba1a79287. Latest consensus state: current_term: 3 leader_uuid: "f73aff31c41749f5a1740f1ba1a79287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } } }
I20260619 16:48:58.466741 18874 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:58.467130 18878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:48:58.467155 18875 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f73aff31c41749f5a1740f1ba1a79287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73aff31c41749f5a1740f1ba1a79287" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34023 } } }
I20260619 16:48:58.467376 18875 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73aff31c41749f5a1740f1ba1a79287 [sys.catalog]: This master's current role is: LEADER
I20260619 16:48:58.472553 18878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:48:58.473320 18878 catalog_manager.cc:1259] Loaded cluster ID: 419a48323e4d427ab5ff31327bb1c857
I20260619 16:48:58.473397 18878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 16:48:58.474184 18878 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe6ae9709ad google::LogMessage::Fail() at ??:0
@ 0x7fe6ae97388a google::LogMessage::SendToLog() at ??:0
@ 0x7fe6ae9704de google::LogMessage::Flush() at ??:0
@ 0x7fe6ae971a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe6b92affdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe6b92b0d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe6b92ad579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe6b92f1640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe6b0bc23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe6b0bc2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe6b0bc43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe6b0bb3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe6b21f76db start_thread at ??:0
@ 0x7fe6ad46e61f clone at ??:0
W20260619 16:48:58.568444 18872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:58.568697 18872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:58.568740 18872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:58.572795 18872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:58.572857 18872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:58.572947 18872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:48:58.578212 18872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:44361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=42785
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:58.579293 18872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:58.580281 18872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:58.590201 18911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.590183 18912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.590210 18914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.590570 18872 server_base.cc:1034] running on GCE node
I20260619 16:48:58.591073 18872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:58.591887 18872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:58.593091 18872 hybrid_clock.cc:648] HybridClock initialized: now 1781887738593055 us; error 47 us; skew 500 ppm
Jun 19 16:48:58 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:58.596582 18872 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:48:58.597509 18872 webserver.cc:460] Webserver started at http://127.3.233.129:42785/ using document root <none> and password file <none>
I20260619 16:48:58.598088 18872 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:58.598150 18872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:58.602236 18872 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:58.604763 18921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.606141 18872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:48:58.606288 18872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "0b6002a1dc884bc7a35ce64f237862c3"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "ba38c04949cc6f508c77b3d9cab79432"
server_key_iv: "9b86760a2f4b91da068631c5f1c2c93d"
server_key_version: "encryptionkey@0"
I20260619 16:48:58.606849 18872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:58.634894 18872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:58.638324 18872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:58.638547 18872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:58.639216 18872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:58.640385 18928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:48:58.646554 18872 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:48:58.646687 18872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260619 16:48:58.646775 18872 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:48:58.648589 18872 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:48:58.648666 18872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 16:48:58.649232 18928 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap starting.
I20260619 16:48:58.659430 18872 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:44361
I20260619 16:48:58.659670 19035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:44361 every 8 connection(s)
I20260619 16:48:58.660497 18872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:48:58.664250 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 18872
I20260619 16:48:58.664747 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:34521
--local_ip_for_outbound_sockets=127.3.233.130
--tserver_master_addrs=127.3.233.190:34023
--webserver_port=42897
--webserver_interface=127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:58.672734 18928 log.cc:824] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:48:58.682986 18928 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:48:58.683434 18928 tablet_bootstrap.cc:492] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Bootstrap complete.
I20260619 16:48:58.684705 18928 ts_tablet_manager.cc:1397] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent bootstrapping tablet: real 0.036s user 0.029s sys 0.004s
I20260619 16:48:58.688174 18928 raft_consensus.cc:348] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.688895 18928 raft_consensus.cc:729] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6002a1dc884bc7a35ce64f237862c3, State: Initialized, Role: FOLLOWER
I20260619 16:48:58.689435 18928 consensus_queue.cc:260] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:48:58.690446 18928 ts_tablet_manager.cc:1428] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260619 16:48:58.714047 19038 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:48:58.662514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:39559)
0619 16:48:58.663036 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.663092 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:48:58.664011 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:48:58.713276 (+ 49265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
W20260619 16:48:58.714660 19036 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:34023 (0 consecutive failures): Network error: Failed to ping master at 127.3.233.190:34023: Client connection negotiation failed: client connection to 127.3.233.190:34023: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:48:58.715394 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.715143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:36211)
0619 16:48:58.715265 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.715338 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:48:58.715993 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.715792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:49691)
0619 16:48:58.715894 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.715938 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:48:58.716151 19036 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:48:58.782889 19039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:58.783214 19039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:58.783259 19039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:58.787546 19039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:58.787613 19039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:58.787708 19039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:48:58.793232 19039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:34521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=42897
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:58.794461 19039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:58.795652 19039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:58.806084 19046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.806123 19047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.806357 19039 server_base.cc:1034] running on GCE node
W20260619 16:48:58.806098 19049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.806882 19039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:58.807597 19039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:58.808794 19039 hybrid_clock.cc:648] HybridClock initialized: now 1781887738808759 us; error 48 us; skew 500 ppm
Jun 19 16:48:58 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:58.812179 19039 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:48:58.812974 19039 webserver.cc:460] Webserver started at http://127.3.233.130:42897/ using document root <none> and password file <none>
I20260619 16:48:58.813490 19039 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:58.813550 19039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:58.817497 19039 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:58.819629 19056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.820899 19039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:48:58.820995 19039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73"
format_stamp: "Formatted at 2026-06-19 16:48:56 on dist-test-slave-cgwx"
server_key: "7935fdd500bffd1089bdaf0743dea55a"
server_key_iv: "6ee614b85bac2b5ac46519ae1e0079fa"
server_key_version: "encryptionkey@0"
I20260619 16:48:58.821610 19039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:58.834648 19039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:58.838042 19039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:58.838263 19039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:58.838860 19039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:58.839782 19039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:58.839856 19039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.839929 19039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:58.839973 19039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:58.849218 19039 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:34521
I20260619 16:48:58.849237 19169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:34521 every 8 connection(s)
I20260619 16:48:58.850255 19039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:48:58.852281 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19039
I20260619 16:48:58.852725 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:45709
--local_ip_for_outbound_sockets=127.3.233.131
--tserver_master_addrs=127.3.233.190:34023
--webserver_port=46047
--webserver_interface=127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:48:58.854624 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.852615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:32809)
0619 16:48:58.853181 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.853636 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":147,"threads_started":1}
W20260619 16:48:58.855167 19170 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:34023 (0 consecutive failures): Network error: Failed to ping master at 127.3.233.190:34023: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
I20260619 16:48:58.855857 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.855561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:43063)
0619 16:48:58.855704 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.855772 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:48:58.856688 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:58.856387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:51311)
0619 16:48:58.856500 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:58.856563 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:48:58.856977 19170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:48:58.967928 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:48:58.968230 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:48:58.968322 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:48:58.972339 19173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:48:58.972424 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:48:58.972506 19173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:48:58.977752 19173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:45709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=46047
--tserver_master_addrs=127.3.233.190:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:48:58.978829 19173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:48:58.979866 19173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:48:58.989850 19179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.989853 19181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:48:58.989862 19178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:48:58.989900 19173 server_base.cc:1034] running on GCE node
I20260619 16:48:58.990576 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:48:58.991297 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:48:58.992506 19173 hybrid_clock.cc:648] HybridClock initialized: now 1781887738992489 us; error 50 us; skew 500 ppm
Jun 19 16:48:58 dist-test-slave-cgwx krb5kdc[17728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:48:58.995572 19173 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:48:58.996531 19173 webserver.cc:460] Webserver started at http://127.3.233.131:46047/ using document root <none> and password file <none>
I20260619 16:48:58.997104 19173 fs_manager.cc:359] Metadata directory not provided
I20260619 16:48:58.997196 19173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:48:59.001226 19173 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:48:59.003885 19188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:59.005447 19173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:48:59.005607 19173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "9d37e4d66cbd4d5b8a6df2045712856e"
format_stamp: "Formatted at 2026-06-19 16:48:57 on dist-test-slave-cgwx"
server_key: "7eb8ee68662e5a2ecba80415c872c450"
server_key_iv: "911cb41ebcc3765cb1e1549f2dd9df97"
server_key_version: "encryptionkey@0"
I20260619 16:48:59.006198 19173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:48:59.018550 19173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:48:59.021943 19173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:48:59.022156 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:48:59.022794 19173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:48:59.023718 19173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:48:59.023788 19173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:59.023861 19173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:48:59.023916 19173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:48:59.033509 19173 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:45709
I20260619 16:48:59.033533 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:45709 every 8 connection(s)
I20260619 16:48:59.034565 19173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:48:59.038582 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:59.036812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:40585)
0619 16:48:59.037294 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:59.037733 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
W20260619 16:48:59.039135 19302 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:34023 (0 consecutive failures): Network error: Failed to ping master at 127.3.233.190:34023: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
I20260619 16:48:59.039737 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:59.039489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:35535)
0619 16:48:59.039637 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:59.039687 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 16:48:59.039876 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19173
I20260619 16:48:59.040488 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:59.040229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:45933)
0619 16:48:59.040373 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:59.040426 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 16:48:59.040683 19302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:48:59.717229 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:59.716799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:44179)
0619 16:48:59.717062 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:59.717122 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260619 16:48:59.858054 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:48:59.857619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:53165)
0619 16:48:59.857890 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 16:48:59.857951 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":69,"threads_started":1}
I20260619 16:49:00.041646 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.041271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:50441)
0619 16:49:00.041495 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:00.041551 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260619 16:49:00.512140 19309 raft_consensus.cc:482] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:00.512377 19309 raft_consensus.cc:504] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:49:00.513517 19309 leader_election.cc:288] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709)
Jun 19 16:49:00 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.130@KRBTEST.COM
Jun 19 16:49:00 dist-test-slave-cgwx krb5kdc[17728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.131@KRBTEST.COM
I20260619 16:49:00.532445 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.513766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:34521 (local address 127.3.233.129:37139)
0619 16:49:00.514070 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:00.514086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:00.514194 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:00.515868 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:00.515873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:00.516404 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:00.516868 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:00.516883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.518143 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.518146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:00.519067 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:00.519076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.520135 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.520138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:00.520303 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:00.525049 (+ 4746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:00.525079 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:00.527316 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:00.529761 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:00.529767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:00.529781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:00.530117 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:00.530523 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:00.530528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:00.530530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:00.530696 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:00.531463 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:00.531469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:00.531698 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 16:49:00.531893 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 16:49:00.532256 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":75,"spinlock_wait_cycles":55808,"thread_start_us":60,"threads_started":1}
I20260619 16:49:00.532459 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.513913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:56105 (local address 127.3.233.131:45709)
0619 16:49:00.514372 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:00.514391 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:00.514422 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:49:00.515162 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:00.515169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:00.515503 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:00.515814 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:00.517044 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.517980 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.519236 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.519816 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.519987 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:00.527854 (+ 7867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:00.527859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:00.527872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:00.527931 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:00.529722 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:00.530315 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:00.530321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:00.530326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:00.530479 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:00.530837 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:00.530841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:00.530842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:00.531354 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:00.531507 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:00.531999 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 16:49:00.532261 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":57,"threads_started":1}
I20260619 16:49:00.532473 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.513915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:37139 (local address 127.3.233.130:34521)
0619 16:49:00.514357 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:00.514371 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:00.514406 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:49:00.515145 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:00.515151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:00.515502 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:00.515814 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:00.517046 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.517991 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.519180 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.519784 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.519962 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:00.527567 (+ 7605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:00.527573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:00.527587 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:00.527617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:00.529581 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:00.530273 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:00.530279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:00.530285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:00.530412 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:00.530830 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:00.530838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:00.530840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:00.531340 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:00.531476 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:00.531998 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 16:49:00.532284 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":117,"threads_started":1}
I20260619 16:49:00.532511 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.513774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:45709 (local address 127.3.233.129:56105)
0619 16:49:00.514124 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:00.514140 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:00.514262 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:00.515867 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:00.515873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:00.516403 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:00.516868 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:00.516882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.518123 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.518127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:00.519073 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:00.519082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:00.519941 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:00.519944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:00.520307 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:00.525049 (+ 4742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:00.525079 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:00.527648 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:00.529850 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:00.529855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:00.529857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:00.530151 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:00.530570 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:00.530573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:00.530575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:00.530709 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:00.531497 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:00.531501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:00.531698 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 16:49:00.531893 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 16:49:00.532343 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":347,"spinlock_wait_cycles":17920,"thread_start_us":76,"threads_started":1}
W20260619 16:49:00.535266 18925 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
W20260619 16:49:00.535769 18922 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
I20260619 16:49:00.535903 18922 leader_election.cc:302] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6002a1dc884bc7a35ce64f237862c3; no voters: 9d37e4d66cbd4d5b8a6df2045712856e, ad3f18f3c6ed42ff89831f1fb15b7c73
I20260619 16:49:00.536170 19309 raft_consensus.cc:2726] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:49:00.718586 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.718078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:49987)
0619 16:49:00.718383 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:00.718478 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260619 16:49:00.859548 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:00.859036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:50953)
0619 16:49:00.859341 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:00.859408 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
I20260619 16:49:01.042786 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:01.042369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:53509)
0619 16:49:01.042616 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:01.042674 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":67,"threads_started":1}
I20260619 16:49:01.719973 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:01.719482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:57059)
0619 16:49:01.719807 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:01.719873 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260619 16:49:01.860803 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:01.860352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:55389)
0619 16:49:01.860632 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:01.860700 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260619 16:49:02.044023 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.043605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:45607)
0619 16:49:02.043859 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.043917 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260619 16:49:02.306075 19319 raft_consensus.cc:482] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:02.306211 19319 raft_consensus.cc:504] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:49:02.306640 19319 leader_election.cc:288] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709)
W20260619 16:49:02.307503 18922 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
W20260619 16:49:02.307657 18925 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
I20260619 16:49:02.307749 18925 leader_election.cc:302] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6002a1dc884bc7a35ce64f237862c3; no voters: 9d37e4d66cbd4d5b8a6df2045712856e, ad3f18f3c6ed42ff89831f1fb15b7c73
I20260619 16:49:02.307900 19319 raft_consensus.cc:2726] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:49:02.721331 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.720870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:35963)
0619 16:49:02.721154 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.721224 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260619 16:49:02.862035 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:02.861622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:46155)
0619 16:49:02.861883 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:02.861943 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260619 16:49:03.045328 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.044855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:60725)
0619 16:49:03.045177 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:03.045238 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260619 16:49:03.722838 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.722339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:36773)
0619 16:49:03.722661 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:03.722736 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":74,"threads_started":1}
I20260619 16:49:03.863348 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.862879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:55331)
0619 16:49:03.863188 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:03.863248 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":136,"threads_started":1}
I20260619 16:49:04.046772 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:04.046302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:54993)
0619 16:49:04.046596 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:04.046659 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":110,"threads_started":1}
I20260619 16:49:04.432332 19326 raft_consensus.cc:482] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:04.432474 19326 raft_consensus.cc:504] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:49:04.432997 19326 leader_election.cc:288] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709)
W20260619 16:49:04.433804 18922 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
W20260619 16:49:04.433936 18925 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
I20260619 16:49:04.434036 18925 leader_election.cc:302] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6002a1dc884bc7a35ce64f237862c3; no voters: 9d37e4d66cbd4d5b8a6df2045712856e, ad3f18f3c6ed42ff89831f1fb15b7c73
I20260619 16:49:04.434193 19326 raft_consensus.cc:2726] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:49:04.724280 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:04.723758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:42085)
0619 16:49:04.724042 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:04.724110 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260619 16:49:04.864777 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:04.864311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:34385)
0619 16:49:04.864596 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:04.864670 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260619 16:49:05.048180 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.047675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:53097)
0619 16:49:05.048002 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.048072 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":129,"threads_started":1}
I20260619 16:49:05.725826 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.725283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:51987)
0619 16:49:05.725583 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.725673 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":119,"threads_started":1}
I20260619 16:49:05.866344 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.865769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:33347)
0619 16:49:05.866157 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.866234 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":90,"threads_started":1}
I20260619 16:49:06.049491 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.049008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:47675)
0619 16:49:06.049328 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.049391 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":78,"threads_started":1}
I20260619 16:49:06.727070 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.726627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:37099)
0619 16:49:06.726884 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.726960 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
I20260619 16:49:06.771168 19334 raft_consensus.cc:482] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:06.771304 19334 raft_consensus.cc:504] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:49:06.771797 19334 leader_election.cc:288] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709)
W20260619 16:49:06.772643 18922 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
W20260619 16:49:06.772790 18925 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
I20260619 16:49:06.772886 18925 leader_election.cc:302] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6002a1dc884bc7a35ce64f237862c3; no voters: 9d37e4d66cbd4d5b8a6df2045712856e, ad3f18f3c6ed42ff89831f1fb15b7c73
I20260619 16:49:06.773046 19334 raft_consensus.cc:2726] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:49:06.867991 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:06.867516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:57609)
0619 16:49:06.867809 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:06.867881 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260619 16:49:07.050745 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.050273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:46397)
0619 16:49:07.050578 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.050650 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260619 16:49:07.728417 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.727923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:33713)
0619 16:49:07.728187 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.728319 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260619 16:49:07.869266 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.868806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:57383)
0619 16:49:07.869099 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.869165 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260619 16:49:08.052109 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.051642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:57469)
0619 16:49:08.051937 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.052004 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260619 16:49:08.729795 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.729332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:36779)
0619 16:49:08.729620 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.729689 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260619 16:49:08.870759 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.870214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:39773)
0619 16:49:08.870581 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.870649 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
I20260619 16:49:09.053351 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.052922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:59295)
0619 16:49:09.053190 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:09.053248 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260619 16:49:09.731071 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.730562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:39733)
0619 16:49:09.730835 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:09.730895 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260619 16:49:09.872152 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:09.871676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:53219)
0619 16:49:09.871987 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:09.872054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260619 16:49:10.054589 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.054145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:43627)
0619 16:49:10.054428 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.054485 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
I20260619 16:49:10.732223 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.731808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:39383)
0619 16:49:10.732076 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.732131 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260619 16:49:10.873624 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.873107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:46229)
0619 16:49:10.873460 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.873527 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":93,"threads_started":1}
I20260619 16:49:11.056058 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.055633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:46715)
0619 16:49:11.055902 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.055964 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260619 16:49:11.606194 19349 raft_consensus.cc:482] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:11.606334 19349 raft_consensus.cc:504] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f18f3c6ed42ff89831f1fb15b7c73" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 34521 } } peers { permanent_uuid: "9d37e4d66cbd4d5b8a6df2045712856e" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 45709 } } peers { permanent_uuid: "0b6002a1dc884bc7a35ce64f237862c3" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 44361 } }
I20260619 16:49:11.606889 19349 leader_election.cc:288] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521), 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709)
W20260619 16:49:11.607823 18922 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d37e4d66cbd4d5b8a6df2045712856e (127.3.233.131:45709): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
W20260619 16:49:11.607946 18925 leader_election.cc:341] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f18f3c6ed42ff89831f1fb15b7c73 (127.3.233.130:34521): Not found: Tablet not found: dee09f4e51c4490baae7475a22dd6302
I20260619 16:49:11.608076 18925 leader_election.cc:302] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b6002a1dc884bc7a35ce64f237862c3; no voters: 9d37e4d66cbd4d5b8a6df2045712856e, ad3f18f3c6ed42ff89831f1fb15b7c73
I20260619 16:49:11.608239 19349 raft_consensus.cc:2726] T dee09f4e51c4490baae7475a22dd6302 P 0b6002a1dc884bc7a35ce64f237862c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:49:11.733599 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.733097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:59873)
0619 16:49:11.733432 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.733497 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":81,"threads_started":1}
I20260619 16:49:11.875092 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.874566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:56121)
0619 16:49:11.874839 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.874995 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260619 16:49:12.057224 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.056790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:46327)
0619 16:49:12.057080 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:12.057133 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":144,"threads_started":1}
I20260619 16:49:12.735111 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.734550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:39335)
0619 16:49:12.734871 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:12.735010 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260619 16:49:12.876328 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.875875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:55161)
0619 16:49:12.876159 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:12.876218 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260619 16:49:13.058569 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.058093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:51755)
0619 16:49:13.058403 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:13.058473 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
I20260619 16:49:13.736642 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.736173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.129:59633)
0619 16:49:13.736478 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:13.736537 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260619 16:49:13.877801 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.877373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.130:38923)
0619 16:49:13.877644 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:13.877705 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260619 16:49:14.044925 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 18872
I20260619 16:49:14.052531 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19039
I20260619 16:49:14.060043 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19173
I20260619 16:49:14.060137 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.059605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34023 (local address 127.3.233.131:40135)
0619 16:49:14.059909 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:14.060023 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":62,"threads_started":1}
2026-06-19T16:49:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19093 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:14 dist-test-slave-cgwx krb5kdc[19361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:14 dist-test-slave-cgwx krb5kdc[19361](info): set up 2 sockets
Jun 19 16:49:14 dist-test-slave-cgwx krb5kdc[19361](info): commencing operation
krb5kdc: starting...
W20260619 16:49:15.115500 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:49:15 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:49:15.242110 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32853
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:32853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:15.356732 19374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:15.357023 19374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:15.357064 19374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:15.361227 19374 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:49:15.361294 19374 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:49:15.361310 19374 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:49:15.361327 19374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:15.361346 19374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:15.361389 19374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:15.361428 19374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:15.361450 19374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:15.366796 19374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:32853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:15.367908 19374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:15.368881 19374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:15.377737 19374 server_base.cc:1034] running on GCE node
W20260619 16:49:15.377774 19379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:15.377899 19382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:15.377799 19380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:15.378593 19374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:15.379799 19374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:15.381008 19374 hybrid_clock.cc:648] HybridClock initialized: now 1781887755380977 us; error 45 us; skew 500 ppm
Jun 19 16:49:15 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:15.383961 19374 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:49:15.384925 19374 webserver.cc:460] Webserver started at http://127.3.233.190:36425/ using document root <none> and password file <none>
I20260619 16:49:15.385512 19374 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:15.385608 19374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:15.385849 19374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:15.388077 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "23d95b27f9344223a31682f1dd95d88e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "4d4e1e2fa6e2a7a7e781aa40362ab31d"
server_key_iv: "5cc8e660545000f77d680404b832eabd"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.388648 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "23d95b27f9344223a31682f1dd95d88e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "4d4e1e2fa6e2a7a7e781aa40362ab31d"
server_key_iv: "5cc8e660545000f77d680404b832eabd"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.392632 19374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:49:15.395560 19389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:15.396914 19374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:15.397073 19374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "23d95b27f9344223a31682f1dd95d88e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "4d4e1e2fa6e2a7a7e781aa40362ab31d"
server_key_iv: "5cc8e660545000f77d680404b832eabd"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.397205 19374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:15.420390 19374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:15.424260 19374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:15.424511 19374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:15.432427 19374 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:32853
I20260619 16:49:15.432758 19441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:32853 every 8 connection(s)
I20260619 16:49:15.433528 19374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:15.436715 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:15.438980 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19374
I20260619 16:49:15.439113 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:49:15.439412 4006 external_mini_cluster.cc:1428] Setting key 676434058cc88d8dcdab806a1c009937
I20260619 16:49:15.443361 19442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Bootstrap starting.
I20260619 16:49:15.445698 19442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:15.446468 19442 log.cc:824] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:15.448638 19442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: No bootstrap required, opened a new log
Jun 19 16:49:15 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:49:15.452299 19442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:15.452564 19442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:15.452632 19442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d95b27f9344223a31682f1dd95d88e, State: Initialized, Role: FOLLOWER
I20260619 16:49:15.453122 19442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:15.453251 19442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:15.453336 19442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:15.453441 19442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:15.454397 19442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:15.454788 19442 leader_election.cc:302] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d95b27f9344223a31682f1dd95d88e; no voters:
I20260619 16:49:15.455135 19442 leader_election.cc:288] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:15.455257 19447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:15.455473 19447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 LEADER]: Becoming Leader. State: Replica: 23d95b27f9344223a31682f1dd95d88e, State: Running, Role: LEADER
I20260619 16:49:15.455636 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:15.441012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.3.233.190:32853)
0619 16:49:15.441546 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:15.441564 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:15.441624 (+ 60us) server_negotiation.cc:407] Connection header received
0619 16:49:15.442515 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:15.442538 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:15.442996 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:15.443385 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:15.444227 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.445225 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.446238 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.446809 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.447015 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:15.449853 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:15.449860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:15.449876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:15.449905 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:15.452189 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:15.452693 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:15.452701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:15.452711 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:15.452837 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:15.453130 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:15.453136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:15.453137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:15.453689 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:15.453908 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:15.454215 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 16:49:15.454568 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":147,"threads_started":1}
I20260619 16:49:15.455843 19447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:15.456528 19442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:15.457605 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d95b27f9344223a31682f1dd95d88e. Latest consensus state: current_term: 1 leader_uuid: "23d95b27f9344223a31682f1dd95d88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } } }
I20260619 16:49:15.458161 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:15.458959 19460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:15.458891 19448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23d95b27f9344223a31682f1dd95d88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } } }
I20260619 16:49:15.459038 19448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:15.461517 19460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:15.467214 19460 catalog_manager.cc:1347] Generated new cluster ID: d235b2ee26714406a97ae4cbd00ed084
I20260619 16:49:15.467293 19460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:15.489840 19460 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:15.491328 19460 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:15.501266 19460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Generated new TSK 0
I20260619 16:49:15.502151 19460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:49:15.532136 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32853
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:15.648437 19468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:15.648774 19468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:15.648850 19468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:15.653128 19468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:15.653230 19468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:15.653324 19468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:49:15.659199 19468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:15.660331 19468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:15.661391 19468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:15.671556 19476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:15.671685 19468 server_base.cc:1034] running on GCE node
W20260619 16:49:15.671574 19473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:15.671571 19474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:15.672264 19468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:15.673043 19468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:15.674269 19468 hybrid_clock.cc:648] HybridClock initialized: now 1781887755674247 us; error 37 us; skew 500 ppm
Jun 19 16:49:15 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:15.677796 19468 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:49:15.678788 19468 webserver.cc:460] Webserver started at http://127.3.233.129:45105/ using document root <none> and password file <none>
I20260619 16:49:15.679438 19468 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:15.679545 19468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:15.679726 19468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:15.682111 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "ebc41f368a23471889f7c3abf244f74e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "f53ba3002c2b6cfd63f481565d7fa7ce"
server_key_iv: "a6f419cc650da2427108f90580f46c89"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.682655 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "ebc41f368a23471889f7c3abf244f74e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "f53ba3002c2b6cfd63f481565d7fa7ce"
server_key_iv: "a6f419cc650da2427108f90580f46c89"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.686628 19468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 16:49:15.689201 19483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:15.690636 19468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:15.690791 19468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "ebc41f368a23471889f7c3abf244f74e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "f53ba3002c2b6cfd63f481565d7fa7ce"
server_key_iv: "a6f419cc650da2427108f90580f46c89"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.691004 19468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:15.715142 19468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:15.718982 19468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:15.719235 19468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:15.719921 19468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:15.720927 19468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:15.720978 19468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:15.721057 19468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:15.721093 19468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:15.730958 19468 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:39145
I20260619 16:49:15.731007 19596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:39145 every 8 connection(s)
I20260619 16:49:15.731930 19468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:15.739557 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19468
I20260619 16:49:15.739706 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:15.740010 4006 external_mini_cluster.cc:1428] Setting key df11892a060146d749deab7c77558de4
Jun 19 16:49:15 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:49:15.746299 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:15.733906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:48901 (local address 127.3.233.190:32853)
0619 16:49:15.734065 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:15.734069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:15.735212 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 16:49:15.736128 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:15.736132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:15.736190 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:15.736269 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:15.737640 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.738190 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.739461 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.739802 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.739923 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:15.742556 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:15.742560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:15.742562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:15.742590 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:15.744181 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:15.744725 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:15.744729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:15.744730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:15.744814 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:15.745165 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:15.745170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:15.745172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:15.745369 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:15.745479 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:15.745992 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 16:49:15.746129 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:15.747203 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:15.734295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:48901)
0619 16:49:15.735058 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:15.735103 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:15.735911 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:15.736441 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:15.736448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:15.736983 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:15.737471 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:15.737483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.738324 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.738328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:15.739319 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:15.739329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.739966 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.739970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:15.740185 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:15.740844 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:15.740860 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:15.742378 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:15.744311 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:15.744317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:15.744334 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:15.744614 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:15.744916 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:15.744919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:15.744921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:15.745067 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:15.745478 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:15.745485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:15.745748 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 16:49:15.745943 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 16:49:15.746279 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":120,"threads_started":1}
I20260619 16:49:15.748510 19597 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32853
I20260619 16:49:15.748783 19597 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:49:15.749423 19597 heartbeater.cc:511] Master 127.3.233.190:32853 requested a full tablet report, sending...
I20260619 16:49:15.751048 19406 ts_manager.cc:194] Registered new tserver with Master: ebc41f368a23471889f7c3abf244f74e (127.3.233.129:39145)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:49:15.752338 19406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:48901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:49:15.766048 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32853
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:15.881686 19602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:15.882004 19602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:15.882054 19602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:15.886294 19602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:15.886377 19602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:15.886479 19602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:49:15.892158 19602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:15.893407 19602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:15.894527 19602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:15.904894 19608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:15.904894 19610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:15.904914 19607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:15.905187 19602 server_base.cc:1034] running on GCE node
I20260619 16:49:15.905777 19602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:15.906571 19602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:15.907801 19602 hybrid_clock.cc:648] HybridClock initialized: now 1781887755907788 us; error 41 us; skew 500 ppm
Jun 19 16:49:15 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:15.911051 19602 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:49:15.912179 19602 webserver.cc:460] Webserver started at http://127.3.233.130:45161/ using document root <none> and password file <none>
I20260619 16:49:15.912789 19602 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:15.912889 19602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:15.913143 19602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:15.915406 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "f851bee8884e44d490d4de3756fa86f6"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "fc199dbb26ef1e6c1b6ded844c7d5185"
server_key_iv: "40f6c981865431a17448fc9a2f2c31aa"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.916044 19602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "f851bee8884e44d490d4de3756fa86f6"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "fc199dbb26ef1e6c1b6ded844c7d5185"
server_key_iv: "40f6c981865431a17448fc9a2f2c31aa"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.920365 19602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 16:49:15.923488 19617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:15.924881 19602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:15.925045 19602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "f851bee8884e44d490d4de3756fa86f6"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "fc199dbb26ef1e6c1b6ded844c7d5185"
server_key_iv: "40f6c981865431a17448fc9a2f2c31aa"
server_key_version: "encryptionkey@0"
I20260619 16:49:15.925181 19602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:15.964874 19602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:15.968766 19602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:15.969031 19602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:15.969719 19602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:15.970696 19602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:15.970777 19602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:15.970849 19602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:15.970882 19602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:15.981931 19602 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:44721
I20260619 16:49:15.981997 19730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:44721 every 8 connection(s)
I20260619 16:49:15.983333 19602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:15.993350 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19602
I20260619 16:49:15.993462 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:15.993746 4006 external_mini_cluster.cc:1428] Setting key d633b7910cc534463147c7ae66577baf
Jun 19 16:49:15 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:49:15.998664 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:15.987183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:33491 (local address 127.3.233.190:32853)
0619 16:49:15.987311 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:15.987317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:15.987388 (+ 71us) server_negotiation.cc:407] Connection header received
0619 16:49:15.988324 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:15.988327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:15.988385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:15.988466 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:15.989653 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.990203 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.991451 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.991695 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.991809 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:15.994492 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:15.994496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:15.994499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:15.994528 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:15.996260 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:15.996837 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:15.996844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:15.996849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:15.996948 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:15.997419 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:15.997422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:15.997423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:15.997599 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:15.997674 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:15.998355 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 16:49:15.998542 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:49:15.999959 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:15.986384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:33491)
0619 16:49:15.987236 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:15.987298 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:15.988112 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:15.988641 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:15.988648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:15.989018 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:15.989504 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:15.989518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.990333 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.990337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:15.991317 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:15.991326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:15.991848 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:15.991853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:15.992031 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:15.992670 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:15.992686 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:15.994334 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:15.996429 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:15.996434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:15.996447 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:15.996729 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:15.997094 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:15.997098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:15.997102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:15.997307 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:15.997703 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:15.997710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:15.998076 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 16:49:15.998309 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 16:49:15.998691 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":143,"threads_started":1}
I20260619 16:49:16.001362 19731 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32853
I20260619 16:49:16.001732 19731 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:16.002303 19731 heartbeater.cc:511] Master 127.3.233.190:32853 requested a full tablet report, sending...
I20260619 16:49:16.003505 19406 ts_manager.cc:194] Registered new tserver with Master: f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:49:16.004385 19406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:33491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:49:16.024096 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32853
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:16.137279 19736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:16.137578 19736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:16.137624 19736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:16.141938 19736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:16.142015 19736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:16.142076 19736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:49:16.147375 19736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:16.148403 19736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:16.149374 19736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:16.159154 19742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.159165 19744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.159165 19741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:16.159622 19736 server_base.cc:1034] running on GCE node
I20260619 16:49:16.160059 19736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:16.160699 19736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:16.161872 19736 hybrid_clock.cc:648] HybridClock initialized: now 1781887756161832 us; error 53 us; skew 500 ppm
Jun 19 16:49:16 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:16.164810 19736 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:49:16.165793 19736 webserver.cc:460] Webserver started at http://127.3.233.131:41821/ using document root <none> and password file <none>
I20260619 16:49:16.166414 19736 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:16.166510 19736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:16.166724 19736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:16.168874 19736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "9f24891cac5e495084b9a02bbec846af"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-cgwx"
server_key: "3ea87705950cfc13962366b452a759f2"
server_key_iv: "e6268c24aa4ec9dd0790c5efd419e854"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.169459 19736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "9f24891cac5e495084b9a02bbec846af"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-cgwx"
server_key: "3ea87705950cfc13962366b452a759f2"
server_key_iv: "e6268c24aa4ec9dd0790c5efd419e854"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.173493 19736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:49:16.176316 19751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.177654 19736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:16.177796 19736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "9f24891cac5e495084b9a02bbec846af"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-cgwx"
server_key: "3ea87705950cfc13962366b452a759f2"
server_key_iv: "e6268c24aa4ec9dd0790c5efd419e854"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.177918 19736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:16.199163 19736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:16.202613 19736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:16.202821 19736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:16.203454 19736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:16.204316 19736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:16.204393 19736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.204460 19736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:16.204509 19736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.214078 19736 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:41825
I20260619 16:49:16.214097 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:41825 every 8 connection(s)
I20260619 16:49:16.215098 19736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:16.221128 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19736
I20260619 16:49:16.221279 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:16.221562 4006 external_mini_cluster.cc:1428] Setting key 14825d2fbf26d639bc094c9e788d73d8
Jun 19 16:49:16 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:49:16.230151 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.217015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:39677 (local address 127.3.233.190:32853)
0619 16:49:16.217193 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.217198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.218266 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 16:49:16.219394 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.219397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.219446 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.219531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.220867 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.221758 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.223281 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.223521 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.223639 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.226177 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.226186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.226188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.226214 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.227849 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.228464 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.228467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.228468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.228548 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.228935 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.228940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.228942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.229177 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.229309 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.229848 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.229986 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:49:16.231115 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.217393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:39677)
0619 16:49:16.218106 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.218153 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.219067 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.219735 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.219743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.220121 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:16.220661 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.220677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.221924 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.221931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.223145 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.223155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.223678 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.223682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.223868 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.224516 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:16.224532 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:16.226000 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:16.228005 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.228011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.228029 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.228298 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.228652 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.228656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.228658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.228810 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.229318 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:16.229327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:16.229608 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.229808 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.230162 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":117,"threads_started":1}
I20260619 16:49:16.232311 19865 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32853
I20260619 16:49:16.232546 19865 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:16.233028 19865 heartbeater.cc:511] Master 127.3.233.190:32853 requested a full tablet report, sending...
I20260619 16:49:16.234036 19406 ts_manager.cc:194] Registered new tserver with Master: 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825)
I20260619 16:49:16.234756 19406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:39677
I20260619 16:49:16.235843 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:16.246531 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.238163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.3.233.190:32853)
0619 16:49:16.238291 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.238295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.238865 (+ 570us) server_negotiation.cc:407] Connection header received
0619 16:49:16.239053 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.239056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.239106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.239198 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.239993 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.240518 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.241392 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.241616 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.241727 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.243286 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.243290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.243297 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.243322 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.244707 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.245183 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.245187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.245189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.245273 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.245600 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.245603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.245605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.245830 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.245980 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.246230 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.246360 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:49:16.249414 19406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:16.252678 19406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:16.269093 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.262754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:41825 (local address 127.0.0.1:35110)
0619 16:49:16.263354 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.263369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.263486 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.264201 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.264204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.264220 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:16.264293 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.264301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.265802 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.265805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.267294 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.267305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.268507 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.268511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.268643 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.268695 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.268805 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.268924 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":52,"thread_start_us":108,"threads_started":1}
I20260619 16:49:16.269668 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.263309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.3.233.130:44721)
0619 16:49:16.263612 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.263620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.263635 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:16.263694 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.263698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.263887 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.264086 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:16.264372 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.265551 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.268131 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.268961 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.269069 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.269203 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.269379 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.269486 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":147,"threads_started":1}
I20260619 16:49:16.270061 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.262404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39145 (local address 127.0.0.1:41578)
0619 16:49:16.262974 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.262989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.263119 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.264577 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.264583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.264613 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:16.264720 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.264726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.266065 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.266069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.267297 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.267305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.269649 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.269653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.269756 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.269788 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.269867 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.269939 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":30,"thread_start_us":101,"threads_started":1}
I20260619 16:49:16.270114 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.263705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.3.233.129:39145)
0619 16:49:16.264035 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.264041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.264062 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:49:16.264132 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.264139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.264332 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.264516 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:16.265059 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.265984 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.268716 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.269486 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.269629 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.269781 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.269916 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.270009 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":114,"threads_started":1}
I20260619 16:49:16.269510 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.261632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:44721 (local address 127.0.0.1:47772)
0619 16:49:16.262049 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.262105 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.262240 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.264075 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.264084 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.264107 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:16.264247 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.264254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.265694 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.265699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.267971 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.267983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.269087 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.269090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.269205 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.269237 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.269313 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.269376 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1036,"thread_start_us":101,"threads_started":1}
I20260619 16:49:16.270495 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.262951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.3.233.131:41825)
0619 16:49:16.263382 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.263392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.263437 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:49:16.263654 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.263660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.263883 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.264043 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:16.264442 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.265650 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.267431 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.268318 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.268497 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.268653 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.269451 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.269565 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":115,"threads_started":1}
I20260619 16:49:16.272898 19531 tablet_service.cc:1467] Processing CreateTablet for tablet 4174312ee87b40e48ce83a045251f17a (DEFAULT_TABLE table=test-table [id=5a8e52b97e354f11bd915b44de55a175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:16.272969 19799 tablet_service.cc:1467] Processing CreateTablet for tablet 4174312ee87b40e48ce83a045251f17a (DEFAULT_TABLE table=test-table [id=5a8e52b97e354f11bd915b44de55a175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:16.272982 19665 tablet_service.cc:1467] Processing CreateTablet for tablet 4174312ee87b40e48ce83a045251f17a (DEFAULT_TABLE table=test-table [id=5a8e52b97e354f11bd915b44de55a175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:16.273922 19531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4174312ee87b40e48ce83a045251f17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:16.273990 19799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4174312ee87b40e48ce83a045251f17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:16.273990 19665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4174312ee87b40e48ce83a045251f17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:16.280542 19882 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Bootstrap starting.
I20260619 16:49:16.281505 19884 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Bootstrap starting.
I20260619 16:49:16.283252 19883 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Bootstrap starting.
I20260619 16:49:16.283542 19882 tablet_bootstrap.cc:654] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:16.283686 19884 tablet_bootstrap.cc:654] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:16.284924 19884 log.cc:824] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:16.285241 19882 log.cc:824] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:16.285887 19883 tablet_bootstrap.cc:654] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:16.286561 19883 log.cc:824] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:16.287190 19884 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: No bootstrap required, opened a new log
I20260619 16:49:16.287406 19884 ts_tablet_manager.cc:1397] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:49:16.288255 19883 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: No bootstrap required, opened a new log
I20260619 16:49:16.288264 19882 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: No bootstrap required, opened a new log
I20260619 16:49:16.288441 19883 ts_tablet_manager.cc:1397] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:49:16.288441 19882 ts_tablet_manager.cc:1397] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 16:49:16.290568 19884 raft_consensus.cc:348] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.290870 19884 raft_consensus.cc:374] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:16.290958 19884 raft_consensus.cc:729] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f24891cac5e495084b9a02bbec846af, State: Initialized, Role: FOLLOWER
I20260619 16:49:16.291458 19884 consensus_queue.cc:260] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.292121 19865 heartbeater.cc:503] Master 127.3.233.190:32853 was elected leader, sending a full tablet report...
I20260619 16:49:16.292518 19884 ts_tablet_manager.cc:1428] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:49:16.292491 19883 raft_consensus.cc:348] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.292757 19883 raft_consensus.cc:374] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:16.292806 19883 raft_consensus.cc:729] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f851bee8884e44d490d4de3756fa86f6, State: Initialized, Role: FOLLOWER
I20260619 16:49:16.292975 19882 raft_consensus.cc:348] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.293236 19882 raft_consensus.cc:374] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:16.293289 19882 raft_consensus.cc:729] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc41f368a23471889f7c3abf244f74e, State: Initialized, Role: FOLLOWER
I20260619 16:49:16.293283 19883 consensus_queue.cc:260] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.293712 19882 consensus_queue.cc:260] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.293951 19731 heartbeater.cc:503] Master 127.3.233.190:32853 was elected leader, sending a full tablet report...
I20260619 16:49:16.294286 19883 ts_tablet_manager.cc:1428] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 16:49:16.294611 19882 ts_tablet_manager.cc:1428] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:49:16.295220 19597 heartbeater.cc:503] Master 127.3.233.190:32853 was elected leader, sending a full tablet report...
W20260619 16:49:16.466117 19866 tablet.cc:2369] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:16.483312 19598 tablet.cc:2369] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:16.484835 19732 tablet.cc:2369] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:16.536054 19890 raft_consensus.cc:482] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:16.536372 19890 raft_consensus.cc:504] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.537676 19890 leader_election.cc:288] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721), 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825)
I20260619 16:49:16.543548 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.538641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:60823 (local address 127.3.233.131:41825)
0619 16:49:16.538847 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.538856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.538954 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:49:16.539053 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.539056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.539105 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.539180 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:16.539457 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.540198 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.542065 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.542680 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.542835 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.542890 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.543327 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.543410 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 16:49:16.543637 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.538096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:39425 (local address 127.3.233.130:44721)
0619 16:49:16.538237 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.538241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.538459 (+ 218us) server_negotiation.cc:407] Connection header received
0619 16:49:16.538651 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.538785 (+ 134us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.538858 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.539009 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:16.539360 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.540190 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.542065 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.542762 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.542867 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.542912 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.543428 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.543506 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:49:16.543663 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.538027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:44721 (local address 127.3.233.129:39425)
0619 16:49:16.538320 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.538347 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.538467 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.539046 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.539051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.539067 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:16.539241 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.539248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.540389 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.540392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.541907 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.541919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.542874 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.542878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.543133 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.543177 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.543266 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.543468 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":293,"thread_start_us":121,"threads_started":1}
I20260619 16:49:16.543891 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.538457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:41825 (local address 127.3.233.129:60823)
0619 16:49:16.538792 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.538807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.538958 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.539229 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.539233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.539245 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:16.539352 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.539358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.540343 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.540346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.541914 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.541919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.542874 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.542878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.543138 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.543180 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.543266 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.543468 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":209,"thread_start_us":111,"threads_started":1}
I20260619 16:49:16.544412 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4174312ee87b40e48ce83a045251f17a" candidate_uuid: "ebc41f368a23471889f7c3abf244f74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f851bee8884e44d490d4de3756fa86f6" is_pre_election: true
I20260619 16:49:16.544597 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4174312ee87b40e48ce83a045251f17a" candidate_uuid: "ebc41f368a23471889f7c3abf244f74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f24891cac5e495084b9a02bbec846af" is_pre_election: true
I20260619 16:49:16.544759 19683 raft_consensus.cc:2454] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc41f368a23471889f7c3abf244f74e in term 0.
I20260619 16:49:16.544961 19819 raft_consensus.cc:2454] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc41f368a23471889f7c3abf244f74e in term 0.
I20260619 16:49:16.545279 19486 leader_election.cc:302] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e, f851bee8884e44d490d4de3756fa86f6; no voters:
I20260619 16:49:16.545572 19890 raft_consensus.cc:2781] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:16.545661 19890 raft_consensus.cc:482] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:16.545706 19890 raft_consensus.cc:3037] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:16.546759 19890 raft_consensus.cc:504] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.547211 19890 leader_election.cc:288] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 1 election: Requested vote from peers f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721), 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825)
I20260619 16:49:16.547603 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4174312ee87b40e48ce83a045251f17a" candidate_uuid: "ebc41f368a23471889f7c3abf244f74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f851bee8884e44d490d4de3756fa86f6"
I20260619 16:49:16.547658 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4174312ee87b40e48ce83a045251f17a" candidate_uuid: "ebc41f368a23471889f7c3abf244f74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f24891cac5e495084b9a02bbec846af"
I20260619 16:49:16.547763 19819 raft_consensus.cc:3037] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:16.547747 19683 raft_consensus.cc:3037] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:16.549088 19683 raft_consensus.cc:2454] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc41f368a23471889f7c3abf244f74e in term 1.
I20260619 16:49:16.549088 19819 raft_consensus.cc:2454] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc41f368a23471889f7c3abf244f74e in term 1.
I20260619 16:49:16.549412 19486 leader_election.cc:302] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e, f851bee8884e44d490d4de3756fa86f6; no voters:
I20260619 16:49:16.549584 19890 raft_consensus.cc:2781] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:16.549813 19890 raft_consensus.cc:686] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 1 LEADER]: Becoming Leader. State: Replica: ebc41f368a23471889f7c3abf244f74e, State: Running, Role: LEADER
I20260619 16:49:16.550195 19890 consensus_queue.cc:237] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:16.553540 19405 catalog_manager.cc:5495] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e reported cstate change: term changed from 0 to 1, leader changed from <none> to ebc41f368a23471889f7c3abf244f74e (127.3.233.129). New cstate: current_term: 1 leader_uuid: "ebc41f368a23471889f7c3abf244f74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:16.567741 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.564076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.3.233.129:39145)
0619 16:49:16.564201 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.564205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.564221 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:49:16.564329 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.564332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.564393 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.564506 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:16.564769 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.565286 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.566193 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.566427 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.566539 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.566864 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:16.567361 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:16.567500 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.567541 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.567597 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:49:16.573769 19819 raft_consensus.cc:1261] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Refusing update from remote peer ebc41f368a23471889f7c3abf244f74e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:16.573788 19683 raft_consensus.cc:1261] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 1 FOLLOWER]: Refusing update from remote peer ebc41f368a23471889f7c3abf244f74e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:16.574505 19890 consensus_queue.cc:1035] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:16.574692 19893 consensus_queue.cc:1035] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:16.591347 19405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:16.591506 19405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:16.594070 19405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Sending DeleteTablet for 3 replicas of tablet 4174312ee87b40e48ce83a045251f17a
I20260619 16:49:16.595082 19799 tablet_service.cc:1514] Processing DeleteTablet for tablet 4174312ee87b40e48ce83a045251f17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:35110
I20260619 16:49:16.595024 19665 tablet_service.cc:1514] Processing DeleteTablet for tablet 4174312ee87b40e48ce83a045251f17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:47772
I20260619 16:49:16.595582 19531 tablet_service.cc:1514] Processing DeleteTablet for tablet 4174312ee87b40e48ce83a045251f17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:41578
I20260619 16:49:16.595656 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19468
I20260619 16:49:16.595649 19906 tablet_replica.cc:330] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: stopping tablet replica
I20260619 16:49:16.596390 19906 raft_consensus.cc:2229] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:16.596765 19906 raft_consensus.cc:2258] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:16.598910 19906 ts_tablet_manager.cc:1905] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:16.599193 19907 tablet_replica.cc:330] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: stopping tablet replica
I20260619 16:49:16.599503 19907 raft_consensus.cc:2229] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:16.599825 19907 raft_consensus.cc:2258] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:16.601255 19907 ts_tablet_manager.cc:1905] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:16.602605 19906 ts_tablet_manager.cc:1918] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:16.602715 19906 log.cc:1196] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/4174312ee87b40e48ce83a045251f17a
I20260619 16:49:16.603093 19906 ts_tablet_manager.cc:1939] T 4174312ee87b40e48ce83a045251f17a P 9f24891cac5e495084b9a02bbec846af: Deleting consensus metadata
I20260619 16:49:16.603816 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19602
W20260619 16:49:16.603861 19392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:39145 (error 108)
I20260619 16:49:16.604957 19392 catalog_manager.cc:4841] TS 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): tablet 4174312ee87b40e48ce83a045251f17a (table test-table [id=5a8e52b97e354f11bd915b44de55a175]) successfully deleted
I20260619 16:49:16.606022 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.605711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39145 (local address 127.0.0.1:41588)
0619 16:49:16.605896 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.605953 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260619 16:49:16.606216 19392 catalog_manager.cc:4584] TS ebc41f368a23471889f7c3abf244f74e (127.3.233.129:39145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4174312ee87b40e48ce83a045251f17a: Network error: Client connection negotiation failed: client connection to 127.3.233.129:39145: connect: Connection refused (error 111)
I20260619 16:49:16.609647 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19736
I20260619 16:49:16.611891 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.611540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:44721 (local address 127.0.0.1:47774)
0619 16:49:16.611712 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.611783 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.130:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 16:49:16.615856 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19374
I20260619 16:49:36.623611 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32853
--webserver_interface=127.3.233.190
--webserver_port=36425
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:32853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:36.738878 19909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:36.739168 19909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:36.739252 19909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:36.743325 19909 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:49:36.743397 19909 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:49:36.743414 19909 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:49:36.743428 19909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:36.743446 19909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:36.743515 19909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:36.743546 19909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:36.743585 19909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:36.749089 19909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:32853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:36.750285 19909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:36.751415 19909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:36.765420 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:36.765460 19909 server_base.cc:1034] running on GCE node
W20260619 16:49:36.765394 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:36.765394 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:36.766583 19909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:36.767879 19909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:36.769068 19909 hybrid_clock.cc:648] HybridClock initialized: now 1781887776769051 us; error 34 us; skew 500 ppm
Jun 19 16:49:36 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:36.772266 19909 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:49:36.773322 19909 webserver.cc:460] Webserver started at http://127.3.233.190:36425/ using document root <none> and password file <none>
I20260619 16:49:36.773960 19909 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:36.774070 19909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:36.778172 19909 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:36.780645 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:36.781949 19909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:36.782099 19909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "23d95b27f9344223a31682f1dd95d88e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "4d4e1e2fa6e2a7a7e781aa40362ab31d"
server_key_iv: "5cc8e660545000f77d680404b832eabd"
server_key_version: "encryptionkey@0"
I20260619 16:49:36.782759 19909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:36.802220 19909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:36.805653 19909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:36.805907 19909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:36.813608 19909 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:32853
I20260619 16:49:36.813902 19977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:32853 every 8 connection(s)
I20260619 16:49:36.814777 19909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:36.819352 19978 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:49:36.820922 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19909
I20260619 16:49:36.821133 19978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Bootstrap starting.
I20260619 16:49:36.821444 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:39145
--local_ip_for_outbound_sockets=127.3.233.129
--tserver_master_addrs=127.3.233.190:32853
--webserver_port=45105
--webserver_interface=127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:36.840457 19978 log.cc:824] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:36.849694 19978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:36.850153 19978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Bootstrap complete.
I20260619 16:49:36.854653 19978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:36.855245 19978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d95b27f9344223a31682f1dd95d88e, State: Initialized, Role: FOLLOWER
I20260619 16:49:36.855804 19978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:36.855930 19978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:36.856032 19978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:36.856159 19978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:49:36.858321 19978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:36.858714 19978 leader_election.cc:302] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d95b27f9344223a31682f1dd95d88e; no voters:
I20260619 16:49:36.859154 19978 leader_election.cc:288] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:49:36.859232 19982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:49:36.859499 19982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 LEADER]: Becoming Leader. State: Replica: 23d95b27f9344223a31682f1dd95d88e, State: Running, Role: LEADER
I20260619 16:49:36.859884 19982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:36.860570 19978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:36.860989 19982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d95b27f9344223a31682f1dd95d88e. Latest consensus state: current_term: 2 leader_uuid: "23d95b27f9344223a31682f1dd95d88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } } }
I20260619 16:49:36.861115 19982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:36.861483 19983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23d95b27f9344223a31682f1dd95d88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } } }
I20260619 16:49:36.861654 19983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:36.861541 19986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:36.867225 19986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:36.867967 19986 catalog_manager.cc:1259] Loaded cluster ID: d235b2ee26714406a97ae4cbd00ed084
I20260619 16:49:36.868041 19986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:36.870503 19986 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:36.871271 19986 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Loaded TSK: 0
I20260619 16:49:36.889209 19986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Generated new TSK 1
I20260619 16:49:36.889504 19986 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:49:36.944772 19980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:36.945042 19980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:36.945087 19980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:36.949205 19980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:36.949268 19980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:36.949359 19980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:49:36.957587 19980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:39145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=45105
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:36.959242 19980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:36.960637 19980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:36.970675 20008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:36.970685 20005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:36.970988 19980 server_base.cc:1034] running on GCE node
W20260619 16:49:36.970700 20006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:36.971505 19980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:36.972246 19980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:36.973443 19980 hybrid_clock.cc:648] HybridClock initialized: now 1781887776973406 us; error 54 us; skew 500 ppm
Jun 19 16:49:36 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:36.976532 19980 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:49:36.977617 19980 webserver.cc:460] Webserver started at http://127.3.233.129:45105/ using document root <none> and password file <none>
I20260619 16:49:36.978227 19980 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:36.978348 19980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:36.982491 19980 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:36.984839 20015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:36.986302 19980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:36.986433 19980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "ebc41f368a23471889f7c3abf244f74e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "f53ba3002c2b6cfd63f481565d7fa7ce"
server_key_iv: "a6f419cc650da2427108f90580f46c89"
server_key_version: "encryptionkey@0"
I20260619 16:49:36.987105 19980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:37.008450 19980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:37.012261 19980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:37.012534 19980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:37.013309 19980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:37.014562 20022 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:49:37.019608 19980 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:49:37.019690 19980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 16:49:37.019778 19980 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:49:37.021621 19980 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:49:37.021690 19980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 16:49:37.021970 20022 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Bootstrap starting.
I20260619 16:49:37.033906 19980 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:39145
I20260619 16:49:37.033921 20129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:39145 every 8 connection(s)
I20260619 16:49:37.035319 19980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:37.038798 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 19980
I20260619 16:49:37.039367 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:44721
--local_ip_for_outbound_sockets=127.3.233.130
--tserver_master_addrs=127.3.233.190:32853
--webserver_port=45161
--webserver_interface=127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:37.046406 20022 log.cc:824] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:49:37 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:49:37.053320 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.037604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:41659 (local address 127.3.233.190:32853)
0619 16:49:37.038021 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.038045 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.038397 (+ 352us) server_negotiation.cc:407] Connection header received
0619 16:49:37.040035 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.040052 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.040477 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.042152 (+ 1675us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.042392 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.043480 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.044671 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.045158 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.045347 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.048160 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.048166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.048179 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.048207 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.050015 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.050576 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.050588 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.050592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.050720 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.051116 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.051120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.051122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.051480 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.051621 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.052171 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.052433 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":185,"threads_started":1}
I20260619 16:49:37.053333 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.037695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:41659)
0619 16:49:37.038235 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.038278 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.039453 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.041043 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.041051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.041441 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:37.041944 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.041958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.043643 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.043647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.044535 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.044544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.045314 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.045317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.045512 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.046154 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:37.046171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:37.047944 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:37.050160 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.050166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.050180 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.050451 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.050819 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.050823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.050825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.051024 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.051604 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:37.051610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:37.051860 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.052036 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.052399 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":167,"threads_started":1}
I20260619 16:49:37.056375 20130 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32853
I20260619 16:49:37.056620 20130 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:37.056979 20022 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:37.057150 20130 heartbeater.cc:511] Master 127.3.233.190:32853 requested a full tablet report, sending...
I20260619 16:49:37.057446 20022 tablet_bootstrap.cc:492] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Bootstrap complete.
I20260619 16:49:37.059054 20022 ts_tablet_manager.cc:1397] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.015s
I20260619 16:49:37.059723 19942 ts_manager.cc:194] Registered new tserver with Master: ebc41f368a23471889f7c3abf244f74e (127.3.233.129:39145)
I20260619 16:49:37.062485 20022 raft_consensus.cc:348] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:37.063090 20022 raft_consensus.cc:729] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc41f368a23471889f7c3abf244f74e, State: Initialized, Role: FOLLOWER
I20260619 16:49:37.063401 19942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:41659
I20260619 16:49:37.063990 20022 consensus_queue.cc:260] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } } peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } }
I20260619 16:49:37.065133 20022 ts_tablet_manager.cc:1428] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260619 16:49:37.065922 20130 heartbeater.cc:503] Master 127.3.233.190:32853 was elected leader, sending a full tablet report...
I20260619 16:49:37.070235 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.063436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.3.233.129:39145)
0619 16:49:37.065671 (+ 2235us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.065683 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.065708 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:49:37.065793 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.065798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.066000 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.066173 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:37.066477 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.067582 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.069027 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.069602 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.069731 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.069872 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.070008 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.070094 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2109,"thread_start_us":105,"threads_started":1}
I20260619 16:49:37.070241 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.062684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39145 (local address 127.0.0.1:33242)
0619 16:49:37.063774 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.063819 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.063941 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.066177 (+ 2236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.066183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.066206 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:37.066341 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.066349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.067722 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.067725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.068886 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.068900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.069725 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.069729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.069833 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.069868 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.069974 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.070098 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":116,"threads_started":1}
I20260619 16:49:37.071549 20064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4174312ee87b40e48ce83a045251f17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:33242
I20260619 16:49:37.072420 20139 tablet_replica.cc:330] stopping tablet replica
I20260619 16:49:37.072719 20139 raft_consensus.cc:2229] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:37.072973 20139 raft_consensus.cc:2258] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:37.073977 20139 ts_tablet_manager.cc:1905] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:37.077646 20139 ts_tablet_manager.cc:1918] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:37.077783 20139 log.cc:1196] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal/wals/4174312ee87b40e48ce83a045251f17a
I20260619 16:49:37.078048 20139 ts_tablet_manager.cc:1939] T 4174312ee87b40e48ce83a045251f17a P ebc41f368a23471889f7c3abf244f74e: Deleting consensus metadata
I20260619 16:49:37.079959 19928 catalog_manager.cc:4841] TS ebc41f368a23471889f7c3abf244f74e (127.3.233.129:39145): tablet 4174312ee87b40e48ce83a045251f17a (table test-table [id=5a8e52b97e354f11bd915b44de55a175]) successfully deleted
W20260619 16:49:37.168921 20134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:37.169188 20134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:37.169265 20134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:37.173507 20134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:37.173589 20134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:37.173697 20134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:49:37.179541 20134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:44721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=45161
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:37.180693 20134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:37.181754 20134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:37.191872 20147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.191865 20145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.191900 20144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.191954 20134 server_base.cc:1034] running on GCE node
I20260619 16:49:37.192487 20134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:37.193197 20134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:37.194412 20134 hybrid_clock.cc:648] HybridClock initialized: now 1781887777194397 us; error 29 us; skew 500 ppm
Jun 19 16:49:37 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:37.197290 20134 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:49:37.198280 20134 webserver.cc:460] Webserver started at http://127.3.233.130:45161/ using document root <none> and password file <none>
I20260619 16:49:37.198884 20134 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:37.199008 20134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:37.202759 20134 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:49:37.205104 20154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.206408 20134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:37.206542 20134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "f851bee8884e44d490d4de3756fa86f6"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "fc199dbb26ef1e6c1b6ded844c7d5185"
server_key_iv: "40f6c981865431a17448fc9a2f2c31aa"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.207142 20134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:37.219340 20134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:37.222582 20134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:37.222820 20134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:37.223477 20134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:37.224589 20161 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 16:49:37.228567 20161 ts_tablet_manager.cc:1852] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 16:49:37.228968 20161 ts_tablet_manager.cc:1905] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:37.232882 20161 ts_tablet_manager.cc:1918] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:37.232981 20161 log.cc:1196] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/4174312ee87b40e48ce83a045251f17a
I20260619 16:49:37.233363 20161 ts_tablet_manager.cc:1939] T 4174312ee87b40e48ce83a045251f17a P f851bee8884e44d490d4de3756fa86f6: Deleting consensus metadata
I20260619 16:49:37.233743 20134 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 16:49:37.233846 20134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260619 16:49:37.233937 20134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:37.233992 20134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.243201 20134 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:44721
I20260619 16:49:37.243240 20268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:44721 every 8 connection(s)
I20260619 16:49:37.244141 20134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:37.248255 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 20134
I20260619 16:49:37.248798 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:41825
--local_ip_for_outbound_sockets=127.3.233.131
--tserver_master_addrs=127.3.233.190:32853
--webserver_port=41821
--webserver_interface=127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:49:37 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:49:37.259739 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.246131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:43715 (local address 127.3.233.190:32853)
0619 16:49:37.246334 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.246338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.247250 (+ 912us) server_negotiation.cc:407] Connection header received
0619 16:49:37.248430 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.248433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.248498 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.248578 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.250261 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.251181 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.252340 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.252856 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.252985 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.255724 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.255727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.255729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.255756 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.257281 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.257989 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.257992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.257993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.258065 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.258494 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.258497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.258498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.258681 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.258761 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.259497 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.259616 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 16:49:37.261013 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.246551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:43715)
0619 16:49:37.247095 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.247139 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.248127 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.248802 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.248821 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.249338 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:37.250053 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.250074 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.251317 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.251320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.252205 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.252213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.253035 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.253037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.253183 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.253832 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:37.253849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:37.255520 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:37.257435 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.257445 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.257465 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.257887 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.258161 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.258165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.258167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.258378 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.258796 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:37.258807 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:37.259217 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.259467 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.259967 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":119,"threads_started":1}
I20260619 16:49:37.262266 20269 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32853
I20260619 16:49:37.262534 20269 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:37.263145 20269 heartbeater.cc:511] Master 127.3.233.190:32853 requested a full tablet report, sending...
I20260619 16:49:37.264256 19942 ts_manager.cc:194] Registered new tserver with Master: f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
I20260619 16:49:37.264842 19942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:43715
W20260619 16:49:37.364012 20272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:37.364310 20272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:37.364400 20272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:37.368578 20272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:37.368732 20272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:37.368836 20272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:49:37.374317 20272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:41825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=41821
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:37.375483 20272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:37.376518 20272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:37.386296 20277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.386359 20280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.386296 20278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.386690 20272 server_base.cc:1034] running on GCE node
I20260619 16:49:37.387203 20272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:37.387981 20272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:37.391266 20272 hybrid_clock.cc:648] HybridClock initialized: now 1781887777391221 us; error 56 us; skew 500 ppm
Jun 19 16:49:37 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:37.394191 20272 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:49:37.395259 20272 webserver.cc:460] Webserver started at http://127.3.233.131:41821/ using document root <none> and password file <none>
I20260619 16:49:37.395879 20272 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:37.395987 20272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:37.399854 20272 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:37.401939 20287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.403281 20272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:37.403443 20272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "9f24891cac5e495084b9a02bbec846af"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-cgwx"
server_key: "3ea87705950cfc13962366b452a759f2"
server_key_iv: "e6268c24aa4ec9dd0790c5efd419e854"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.403982 20272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:37.427774 20272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:37.431226 20272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:37.431447 20272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:37.432078 20272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:37.432924 20272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:37.432996 20272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.433070 20272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:37.433115 20272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.442653 20272 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:41825
I20260619 16:49:37.442675 20400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:41825 every 8 connection(s)
I20260619 16:49:37.443682 20272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:37.446373 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 20272
Jun 19 16:49:37 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:49:37.456694 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.447087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.3.233.190:32853)
0619 16:49:37.447359 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.447363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.447546 (+ 183us) server_negotiation.cc:407] Connection header received
0619 16:49:37.447658 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.447661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.447711 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.447801 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.448684 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.449489 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.450726 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.451091 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.451202 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.452864 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.452868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.452871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.452898 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.454614 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.455256 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.455260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.455262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.455364 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.455751 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.455755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.455757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.455956 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.456192 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.456370 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.456508 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260619 16:49:37.459523 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.445549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:52131 (local address 127.3.233.190:32853)
0619 16:49:37.445720 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.445725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.446905 (+ 1180us) server_negotiation.cc:407] Connection header received
0619 16:49:37.448109 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.448112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.448166 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.448264 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.449710 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.450485 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.452064 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.452482 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.452594 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.455606 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.455610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.455612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.455639 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.457374 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.457942 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.457945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.457947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.458028 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.458390 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.458393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.458395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.458562 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.458684 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.459277 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.459384 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:49:37.460484 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.446029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:52131)
0619 16:49:37.446725 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.446776 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.447892 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.448416 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.448423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.448927 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:37.449560 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.449580 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.450616 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.450618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.451947 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.451956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.452637 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.452639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.452809 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.453469 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:37.453494 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:37.455413 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:37.457485 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.457492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.457509 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.457857 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.458126 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.458129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.458131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.458288 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.458687 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:37.458697 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:37.459032 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.459224 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.459587 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":134,"threads_started":1}
I20260619 16:49:37.461623 20401 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32853
I20260619 16:49:37.461849 20401 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:37.462289 20401 heartbeater.cc:511] Master 127.3.233.190:32853 requested a full tablet report, sending...
I20260619 16:49:37.463408 19942 ts_manager.cc:194] Registered new tserver with Master: 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825)
I20260619 16:49:37.464026 19942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:52131
I20260619 16:49:37.470359 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:37.474500 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.471094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.3.233.190:32853)
0619 16:49:37.471271 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.471275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.471448 (+ 173us) server_negotiation.cc:407] Connection header received
0619 16:49:37.471587 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.471589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.471686 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.471778 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:37.471997 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.472524 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.473418 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.473642 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.473733 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.474038 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:37.474175 (+ 137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 16:49:37.474400 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50040: token expired
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 16:49:37.474689 20132 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50040: token expired
W20260619 16:49:37.475052 19871 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.233.190:32853)
I20260619 16:49:37.482725 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.475476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.3.233.190:32853)
0619 16:49:37.475595 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.475599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.475617 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:49:37.475700 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.475702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.475738 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.475807 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.476426 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.476951 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.477745 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.477994 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.478074 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.479522 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.479526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.479528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.479561 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.481139 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.481569 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.481572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.481574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.481660 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.481940 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.481942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.481943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.482148 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.482250 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.482504 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.482609 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:49:37.485316 19942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:37.488953 19942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:37.497632 20064 tablet_service.cc:1467] Processing CreateTablet for tablet 35e2825c846049a69b83fe5890c03e82 (DEFAULT_TABLE table=test-table [id=05da1478a93541659bbbc6d68bc41cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:37.498369 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e2825c846049a69b83fe5890c03e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:37.501586 20022 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Bootstrap starting.
I20260619 16:49:37.502080 20410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.496617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:44721 (local address 127.0.0.1:42380)
0619 16:49:37.496907 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.496925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.497024 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.497832 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.497839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.497869 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:37.498042 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.498051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.499211 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.499215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.500054 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.500062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.501518 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.501523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.501629 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.501763 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.501839 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.501923 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":47,"spinlock_wait_cycles":768,"thread_start_us":132,"threads_started":1}
I20260619 16:49:37.502543 20022 tablet_bootstrap.cc:654] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:37.502612 20411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.496768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.3.233.131:41825)
0619 16:49:37.497292 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.497305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.497330 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:49:37.497421 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.497427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.497628 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.498285 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:37.498563 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.499748 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.500972 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.501681 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.502148 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.502281 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.502335 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.502429 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":87,"threads_started":1}
I20260619 16:49:37.502914 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.496817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.3.233.130:44721)
0619 16:49:37.497230 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.497241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.497262 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:49:37.497327 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.497331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.497500 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.497664 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:37.498175 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.499102 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.500199 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.501135 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.501276 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.501443 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.502616 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.502738 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":98,"threads_started":1}
I20260619 16:49:37.502265 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.496361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:41825 (local address 127.0.0.1:49492)
0619 16:49:37.496493 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.496509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.496588 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.497810 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.497814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.497835 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:37.497974 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.497982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.499923 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.499926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.500843 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.500851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.501812 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.501816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.501899 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.501922 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.501971 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.502041 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":3,"spinlock_wait_cycles":896}
I20260619 16:49:37.503659 20022 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: No bootstrap required, opened a new log
I20260619 16:49:37.503790 20022 ts_tablet_manager.cc:1397] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:37.504415 20022 raft_consensus.cc:348] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.504540 20022 raft_consensus.cc:374] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:37.504585 20022 raft_consensus.cc:729] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc41f368a23471889f7c3abf244f74e, State: Initialized, Role: FOLLOWER
I20260619 16:49:37.504715 20022 consensus_queue.cc:260] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.504917 20203 tablet_service.cc:1467] Processing CreateTablet for tablet 35e2825c846049a69b83fe5890c03e82 (DEFAULT_TABLE table=test-table [id=05da1478a93541659bbbc6d68bc41cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:37.505815 20022 ts_tablet_manager.cc:1428] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:37.506160 20203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e2825c846049a69b83fe5890c03e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:37.507175 20335 tablet_service.cc:1467] Processing CreateTablet for tablet 35e2825c846049a69b83fe5890c03e82 (DEFAULT_TABLE table=test-table [id=05da1478a93541659bbbc6d68bc41cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:37.508198 20335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e2825c846049a69b83fe5890c03e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:37.511149 20161 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Bootstrap starting.
I20260619 16:49:37.513394 20161 tablet_bootstrap.cc:654] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:37.514259 20161 log.cc:824] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:37.515853 20415 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Bootstrap starting.
I20260619 16:49:37.516407 20161 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: No bootstrap required, opened a new log
I20260619 16:49:37.516587 20161 ts_tablet_manager.cc:1397] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:49:37.518378 20415 tablet_bootstrap.cc:654] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:37.519164 20415 log.cc:824] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:37.519620 20161 raft_consensus.cc:348] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.519913 20161 raft_consensus.cc:374] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:37.519985 20161 raft_consensus.cc:729] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f851bee8884e44d490d4de3756fa86f6, State: Initialized, Role: FOLLOWER
I20260619 16:49:37.520466 20161 consensus_queue.cc:260] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.521130 20269 heartbeater.cc:503] Master 127.3.233.190:32853 was elected leader, sending a full tablet report...
I20260619 16:49:37.521238 20415 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: No bootstrap required, opened a new log
I20260619 16:49:37.521478 20161 ts_tablet_manager.cc:1428] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:49:37.521636 20415 ts_tablet_manager.cc:1397] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:49:37.523343 20417 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:37.523488 20417 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.524619 20417 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), ebc41f368a23471889f7c3abf244f74e (127.3.233.129:39145)
I20260619 16:49:37.525887 20415 raft_consensus.cc:348] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.526139 20415 raft_consensus.cc:374] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:37.526191 20415 raft_consensus.cc:729] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f24891cac5e495084b9a02bbec846af, State: Initialized, Role: FOLLOWER
I20260619 16:49:37.526616 20415 consensus_queue.cc:260] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.527261 20401 heartbeater.cc:503] Master 127.3.233.190:32853 was elected leader, sending a full tablet report...
I20260619 16:49:37.527652 20415 ts_tablet_manager.cc:1428] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:49:37.529963 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.524822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:41825 (local address 127.3.233.130:51203)
0619 16:49:37.524957 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.524975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.525058 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.525863 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.525866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.525883 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:37.525996 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.526002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.527054 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.527058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.528549 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.528560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.529525 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.529529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.529650 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.529672 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.529747 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.529820 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":106}
I20260619 16:49:37.529987 20411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.525434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:51203 (local address 127.3.233.131:41825)
0619 16:49:37.525560 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.525566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.525581 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:37.525624 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.525627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.525682 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.525749 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:37.526117 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.526900 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.528809 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.529408 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.529528 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.529574 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.529778 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.529876 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:49:37.530088 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.525223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:52507 (local address 127.3.233.129:39145)
0619 16:49:37.525453 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.525459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.525475 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:49:37.525641 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.525645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.525772 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.525819 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:37.526186 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.526952 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.528859 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.529414 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.529518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.529568 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.529901 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.529972 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260619 16:49:37.530378 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.525064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39145 (local address 127.3.233.130:52507)
0619 16:49:37.525399 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.525412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.525494 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.525938 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.525941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.525954 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:37.526071 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.526079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.527073 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.527076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.528710 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.528717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.529651 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.529653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.529769 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.529787 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.530202 (+ 415us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.530275 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":261,"thread_start_us":111,"threads_started":1}
I20260619 16:49:37.530969 20084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e2825c846049a69b83fe5890c03e82" candidate_uuid: "f851bee8884e44d490d4de3756fa86f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc41f368a23471889f7c3abf244f74e" is_pre_election: true
I20260619 16:49:37.531147 20355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e2825c846049a69b83fe5890c03e82" candidate_uuid: "f851bee8884e44d490d4de3756fa86f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f24891cac5e495084b9a02bbec846af" is_pre_election: true
I20260619 16:49:37.531298 20084 raft_consensus.cc:2454] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f851bee8884e44d490d4de3756fa86f6 in term 0.
I20260619 16:49:37.531425 20355 raft_consensus.cc:2454] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f851bee8884e44d490d4de3756fa86f6 in term 0.
I20260619 16:49:37.531875 20157 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6; no voters:
I20260619 16:49:37.532100 20417 raft_consensus.cc:2781] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:37.532184 20417 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:37.532229 20417 raft_consensus.cc:3037] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:37.533249 20417 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:37.533572 20417 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [CANDIDATE]: Term 1 election: Requested vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), ebc41f368a23471889f7c3abf244f74e (127.3.233.129:39145)
I20260619 16:49:37.533913 20355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e2825c846049a69b83fe5890c03e82" candidate_uuid: "f851bee8884e44d490d4de3756fa86f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f24891cac5e495084b9a02bbec846af"
I20260619 16:49:37.533968 20084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e2825c846049a69b83fe5890c03e82" candidate_uuid: "f851bee8884e44d490d4de3756fa86f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc41f368a23471889f7c3abf244f74e"
I20260619 16:49:37.534044 20355 raft_consensus.cc:3037] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:37.534114 20084 raft_consensus.cc:3037] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:37.535100 20355 raft_consensus.cc:2454] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f851bee8884e44d490d4de3756fa86f6 in term 1.
I20260619 16:49:37.535367 20084 raft_consensus.cc:2454] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f851bee8884e44d490d4de3756fa86f6 in term 1.
I20260619 16:49:37.535403 20157 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6; no voters:
I20260619 16:49:37.535553 20417 raft_consensus.cc:2781] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:37.535774 20417 raft_consensus.cc:686] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 1 LEADER]: Becoming Leader. State: Replica: f851bee8884e44d490d4de3756fa86f6, State: Running, Role: LEADER
I20260619 16:49:37.536094 20417 consensus_queue.cc:237] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
W20260619 16:49:37.536625 20131 tablet.cc:2369] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:37.539330 19942 catalog_manager.cc:5495] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f851bee8884e44d490d4de3756fa86f6 (127.3.233.130). New cstate: current_term: 1 leader_uuid: "f851bee8884e44d490d4de3756fa86f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:37.556610 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.552838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.3.233.130:44721)
0619 16:49:37.552970 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.552973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.552986 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:49:37.553079 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.553081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.553136 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.553221 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:37.553520 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.554056 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.555066 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.555330 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.555422 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.555773 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:37.556227 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:37.556349 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.556415 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.556471 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:49:37.563066 20355 raft_consensus.cc:1261] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Refusing update from remote peer f851bee8884e44d490d4de3756fa86f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:37.563154 20084 raft_consensus.cc:1261] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Refusing update from remote peer f851bee8884e44d490d4de3756fa86f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:37.563777 20417 consensus_queue.cc:1035] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:37.564024 20420 consensus_queue.cc:1035] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:37.570799 20424 mvcc.cc:205] Tried to move back new op lower bound from 7298612336889905152 to 7298612336790704128. Current Snapshot: MvccSnapshot[applied={T|T < 7298612336889905152}]
I20260619 16:49:37.571643 20423 mvcc.cc:205] Tried to move back new op lower bound from 7298612336889905152 to 7298612336790704128. Current Snapshot: MvccSnapshot[applied={T|T < 7298612336889905152}]
I20260619 16:49:37.572412 20426 mvcc.cc:205] Tried to move back new op lower bound from 7298612336889905152 to 7298612336790704128. Current Snapshot: MvccSnapshot[applied={T|T < 7298612336889905152}]
I20260619 16:49:37.580837 19942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:37.581028 19942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:37.583681 19942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Sending DeleteTablet for 3 replicas of tablet 35e2825c846049a69b83fe5890c03e82
I20260619 16:49:37.584406 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19980
I20260619 16:49:37.584446 20064 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e2825c846049a69b83fe5890c03e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:37 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:33242
I20260619 16:49:37.584540 20335 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e2825c846049a69b83fe5890c03e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:37 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:49492
I20260619 16:49:37.585175 20203 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e2825c846049a69b83fe5890c03e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:37 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:42380
I20260619 16:49:37.585467 20432 tablet_replica.cc:330] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: stopping tablet replica
I20260619 16:49:37.585862 20432 raft_consensus.cc:2229] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:37.587378 20432 raft_consensus.cc:2258] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:37.587785 20433 tablet_replica.cc:330] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: stopping tablet replica
I20260619 16:49:37.588090 20433 raft_consensus.cc:2229] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:37.588510 20433 raft_consensus.cc:2258] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:37.589053 20432 ts_tablet_manager.cc:1905] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:37.590106 20433 ts_tablet_manager.cc:1905] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:37.591998 20432 ts_tablet_manager.cc:1918] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:37.592166 20432 log.cc:1196] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/35e2825c846049a69b83fe5890c03e82
I20260619 16:49:37.592571 20432 ts_tablet_manager.cc:1939] T 35e2825c846049a69b83fe5890c03e82 P 9f24891cac5e495084b9a02bbec846af: Deleting consensus metadata
W20260619 16:49:37.592572 19928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:39145 (error 108)
I20260619 16:49:37.592837 20433 ts_tablet_manager.cc:1918] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:37.592922 20433 log.cc:1196] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/35e2825c846049a69b83fe5890c03e82
I20260619 16:49:37.593187 20433 ts_tablet_manager.cc:1939] T 35e2825c846049a69b83fe5890c03e82 P f851bee8884e44d490d4de3756fa86f6: Deleting consensus metadata
I20260619 16:49:37.593726 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 20134
I20260619 16:49:37.595773 19928 catalog_manager.cc:4841] TS 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): tablet 35e2825c846049a69b83fe5890c03e82 (table test-table [id=05da1478a93541659bbbc6d68bc41cd2]) successfully deleted
I20260619 16:49:37.597909 20410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.596221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:39145 (local address 127.0.0.1:33244)
0619 16:49:37.597714 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.597800 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1373}
W20260619 16:49:37.598076 19928 catalog_manager.cc:4584] TS ebc41f368a23471889f7c3abf244f74e (127.3.233.129:39145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35e2825c846049a69b83fe5890c03e82: Network error: Client connection negotiation failed: client connection to 127.3.233.129:39145: connect: Connection refused (error 111)
I20260619 16:49:37.600389 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 20272
I20260619 16:49:37.601511 20410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.600635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:44721 (local address 127.0.0.1:42402)
0619 16:49:37.601339 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.601409 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.130:44721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606}
I20260619 16:49:37.606221 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 19909
I20260619 16:49:57.612879 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32853
--webserver_interface=127.3.233.190
--webserver_port=36425
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:32853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:57.730266 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:57.730638 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:57.730715 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:57.734807 20435 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:49:57.734889 20435 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:49:57.734913 20435 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:49:57.734937 20435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:57.734997 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:57.735059 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:57.735105 20435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:57.735143 20435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:57.740764 20435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:32853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:57.741891 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:57.742969 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:57.752385 20435 server_base.cc:1034] running on GCE node
W20260619 16:49:57.752369 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:57.752369 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:57.752632 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:57.753309 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:57.754451 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:57.755615 20435 hybrid_clock.cc:648] HybridClock initialized: now 1781887797755600 us; error 32 us; skew 500 ppm
Jun 19 16:49:57 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:57.758560 20435 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:49:57.759609 20435 webserver.cc:460] Webserver started at http://127.3.233.190:36425/ using document root <none> and password file <none>
I20260619 16:49:57.760237 20435 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:57.760331 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:57.764256 20435 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:57.766587 20450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:57.767901 20435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:57.768043 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "23d95b27f9344223a31682f1dd95d88e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "4d4e1e2fa6e2a7a7e781aa40362ab31d"
server_key_iv: "5cc8e660545000f77d680404b832eabd"
server_key_version: "encryptionkey@0"
I20260619 16:49:57.768659 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:57.780880 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:57.784397 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:57.784610 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:57.792145 20435 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:32853
I20260619 16:49:57.792419 20502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:32853 every 8 connection(s)
I20260619 16:49:57.793391 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:57.797916 20503 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:49:57.799990 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Bootstrap starting.
I20260619 16:49:57.800697 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 20435
I20260619 16:49:57.801261 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:39145
--local_ip_for_outbound_sockets=127.3.233.129
--tserver_master_addrs=127.3.233.190:32853
--webserver_port=45105
--webserver_interface=127.3.233.129
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:57.821106 20503 log.cc:824] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:57.833766 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:57.834270 20503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Bootstrap complete.
I20260619 16:49:57.839221 20503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:57.839658 20503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d95b27f9344223a31682f1dd95d88e, State: Initialized, Role: FOLLOWER
I20260619 16:49:57.840258 20503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:57.840387 20503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:57.840483 20503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:57.840622 20503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:49:57.842734 20503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:57.843168 20503 leader_election.cc:302] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d95b27f9344223a31682f1dd95d88e; no voters:
I20260619 16:49:57.843550 20503 leader_election.cc:288] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:49:57.843645 20507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:49:57.843887 20507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [term 3 LEADER]: Becoming Leader. State: Replica: 23d95b27f9344223a31682f1dd95d88e, State: Running, Role: LEADER
I20260619 16:49:57.844220 20507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } }
I20260619 16:49:57.844993 20503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:57.845786 20508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d95b27f9344223a31682f1dd95d88e. Latest consensus state: current_term: 3 leader_uuid: "23d95b27f9344223a31682f1dd95d88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } } }
I20260619 16:49:57.845983 20508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:57.847061 20516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:57.847232 20509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "23d95b27f9344223a31682f1dd95d88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d95b27f9344223a31682f1dd95d88e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32853 } } }
I20260619 16:49:57.847376 20509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:57.852953 20516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:57.853644 20516 catalog_manager.cc:1259] Loaded cluster ID: d235b2ee26714406a97ae4cbd00ed084
I20260619 16:49:57.853713 20516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:57.856691 20516 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:57.857542 20516 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Loaded TSK: 0
I20260619 16:49:57.857630 20516 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23d95b27f9344223a31682f1dd95d88e: Loaded TSK: 1
F20260619 16:49:57.858366 20516 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc5210429ad google::LogMessage::Fail() at ??:0
@ 0x7fc52104588a google::LogMessage::SendToLog() at ??:0
@ 0x7fc5210424de google::LogMessage::Flush() at ??:0
@ 0x7fc521043a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc523d0618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc523cfef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc52b9a77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc52b98199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc52b982178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc52b9c3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc52b9e282b std::function<>::operator()() at ??:0
@ 0x7fc52b981b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc52b982f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc52b97f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc52b9c3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5232943d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc523294c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc5232963f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc523285512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc5248c96db start_thread at ??:0
@ 0x7fc51fb4061f clone at ??:0
W20260619 16:49:57.953269 20505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:57.953670 20505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:57.953739 20505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:57.960008 20505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:57.960101 20505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:57.960193 20505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:49:57.968582 20505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:39145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=45105
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:57.970031 20505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:57.971396 20505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:57.983893 20550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:57.983938 20551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:57.984144 20505 server_base.cc:1034] running on GCE node
W20260619 16:49:57.984324 20553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:57.984771 20505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:57.985569 20505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:57.986766 20505 hybrid_clock.cc:648] HybridClock initialized: now 1781887797986713 us; error 61 us; skew 500 ppm
Jun 19 16:49:57 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:57.989934 20505 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:49:57.990876 20505 webserver.cc:460] Webserver started at http://127.3.233.129:45105/ using document root <none> and password file <none>
I20260619 16:49:57.991472 20505 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:57.991533 20505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:57.995615 20505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:57.997985 20560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:57.999539 20505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:57.999693 20505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "ebc41f368a23471889f7c3abf244f74e"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "f53ba3002c2b6cfd63f481565d7fa7ce"
server_key_iv: "a6f419cc650da2427108f90580f46c89"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.000267 20505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:58.017972 20505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:58.021590 20505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:58.021812 20505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:58.022436 20505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:58.023674 20567 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:49:58.029261 20505 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:49:58.029379 20505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 16:49:58.029454 20505 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:49:58.031703 20505 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:49:58.031755 20505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:58.039534 20567 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Bootstrap starting.
I20260619 16:49:58.042840 20505 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:39145
I20260619 16:49:58.043984 20505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:58.045583 20674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:39145 every 8 connection(s)
I20260619 16:49:58.049443 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 20505
I20260619 16:49:58.049950 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:44721
--local_ip_for_outbound_sockets=127.3.233.130
--tserver_master_addrs=127.3.233.190:32853
--webserver_port=45161
--webserver_interface=127.3.233.130
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:58.060061 20567 log.cc:824] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:58.069928 20567 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:58.070442 20567 tablet_bootstrap.cc:492] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Bootstrap complete.
I20260619 16:49:58.071900 20567 ts_tablet_manager.cc:1397] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.009s
I20260619 16:49:58.075265 20567 raft_consensus.cc:348] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:58.075882 20567 raft_consensus.cc:729] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc41f368a23471889f7c3abf244f74e, State: Initialized, Role: FOLLOWER
I20260619 16:49:58.076416 20567 consensus_queue.cc:260] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:58.077503 20567 ts_tablet_manager.cc:1428] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 16:49:58.102283 20677 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:58.046080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:36099)
0619 16:49:58.046622 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.046685 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:58.047734 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:58.101444 (+ 53710us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":130,"threads_started":1}
W20260619 16:49:58.102993 20675 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:32853 (0 consecutive failures): Network error: Failed to ping master at 127.3.233.190:32853: Client connection negotiation failed: client connection to 127.3.233.190:32853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:49:58.103812 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.103529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:39253)
0619 16:49:58.103659 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.103743 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:49:58.104532 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.104317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:56519)
0619 16:49:58.104423 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.104471 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:49:58.104744 20675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:49:58.179584 20678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:58.179899 20678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:58.179989 20678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:58.184595 20678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:58.184706 20678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:58.184806 20678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:49:58.190677 20678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:44721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=45161
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:58.191874 20678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:58.192940 20678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:58.203284 20678 server_base.cc:1034] running on GCE node
W20260619 16:49:58.203208 20688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.203203 20685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.203203 20686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.203936 20678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:58.204684 20678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:58.205897 20678 hybrid_clock.cc:648] HybridClock initialized: now 1781887798205854 us; error 55 us; skew 500 ppm
Jun 19 16:49:58 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:58.209143 20678 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:49:58.210183 20678 webserver.cc:460] Webserver started at http://127.3.233.130:45161/ using document root <none> and password file <none>
I20260619 16:49:58.210799 20678 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:58.210894 20678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:58.215227 20678 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 16:49:58.217700 20695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.219296 20678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:58.219453 20678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "f851bee8884e44d490d4de3756fa86f6"
format_stamp: "Formatted at 2026-06-19 16:49:15 on dist-test-slave-cgwx"
server_key: "fc199dbb26ef1e6c1b6ded844c7d5185"
server_key_iv: "40f6c981865431a17448fc9a2f2c31aa"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.220135 20678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:58.250165 20678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:58.253701 20678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:58.253928 20678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:58.254588 20678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:58.255505 20678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:58.255585 20678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.255651 20678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:58.255700 20678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.265272 20678 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:44721
I20260619 16:49:58.265290 20808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:44721 every 8 connection(s)
I20260619 16:49:58.266358 20678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:58.270536 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.268784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:39143)
0619 16:49:58.269537 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.269667 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":123,"threads_started":1}
I20260619 16:49:58.270845 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 20678
W20260619 16:49:58.271238 20809 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:32853 (0 consecutive failures): Network error: Failed to ping master at 127.3.233.190:32853: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
I20260619 16:49:58.271476 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:41825
--local_ip_for_outbound_sockets=127.3.233.131
--tserver_master_addrs=127.3.233.190:32853
--webserver_port=41821
--webserver_interface=127.3.233.131
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:58.271986 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.271719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:38683)
0619 16:49:58.271850 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.271903 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:49:58.272683 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.272463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:32871)
0619 16:49:58.272587 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.272635 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:49:58.272879 20809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:49:58.384021 20812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:58.384334 20812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:58.384419 20812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:58.388514 20812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:58.388616 20812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:58.388713 20812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:49:58.394136 20812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:41825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=41821
--tserver_master_addrs=127.3.233.190:32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:49:58.395324 20812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:58.396405 20812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:58.406829 20818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.406872 20820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.406829 20817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.407203 20812 server_base.cc:1034] running on GCE node
I20260619 16:49:58.407729 20812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:58.408434 20812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:58.409639 20812 hybrid_clock.cc:648] HybridClock initialized: now 1781887798409595 us; error 59 us; skew 500 ppm
Jun 19 16:49:58 dist-test-slave-cgwx krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:58.412640 20812 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:49:58.413551 20812 webserver.cc:460] Webserver started at http://127.3.233.131:41821/ using document root <none> and password file <none>
I20260619 16:49:58.414119 20812 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:58.414181 20812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:58.418157 20812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:58.420473 20827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.422365 20812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:58.422528 20812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "9f24891cac5e495084b9a02bbec846af"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-cgwx"
server_key: "3ea87705950cfc13962366b452a759f2"
server_key_iv: "e6268c24aa4ec9dd0790c5efd419e854"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.423288 20812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:58.447636 20812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:58.451293 20812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:58.451543 20812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:58.452198 20812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:58.453220 20812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:58.453296 20812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.453372 20812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:58.453418 20812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.462612 20812 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:41825
I20260619 16:49:58.462677 20940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:41825 every 8 connection(s)
I20260619 16:49:58.463806 20812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:58.467995 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.466045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:55233)
0619 16:49:58.466555 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.467176 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
W20260619 16:49:58.468564 20941 heartbeater.cc:644] Failed to heartbeat to 127.3.233.190:32853 (0 consecutive failures): Network error: Failed to ping master at 127.3.233.190:32853: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
I20260619 16:49:58.468776 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 20812
I20260619 16:49:58.469396 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.469091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:33755)
0619 16:49:58.469255 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.469327 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:49:58.470139 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.469918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:56701)
0619 16:49:58.470037 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.470085 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:49:58.470296 20941 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:49:59.105858 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.105383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:34893)
0619 16:49:59.105677 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.105738 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260619 16:49:59.273895 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.273462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:35881)
0619 16:49:59.273729 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.273790 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
I20260619 16:49:59.471284 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.470855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:33193)
0619 16:49:59.471130 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.471188 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
I20260619 16:49:59.862963 20948 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:59.863222 20948 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:49:59.864413 20948 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
Jun 19 16:49:59 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.131@KRBTEST.COM
Jun 19 16:49:59 dist-test-slave-cgwx krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.130@KRBTEST.COM
I20260619 16:49:59.884083 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.865126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:44721 (local address 127.3.233.129:46321)
0619 16:49:59.865420 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.865430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.865503 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.866996 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.866999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.867377 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:59.867870 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.867886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.869155 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.869159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.870083 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.870093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.870835 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.870839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.871109 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.876612 (+ 5503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:59.876646 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:59.878850 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:59.881377 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:59.881380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:59.881381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:59.881829 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:59.882249 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:59.882251 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:59.882253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:59.882331 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:59.883086 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:59.883089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:59.883310 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.883500 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.883882 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":109,"spinlock_wait_cycles":37120,"thread_start_us":57,"threads_started":1}
I20260619 16:49:59.884094 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.864865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:57173 (local address 127.3.233.131:41825)
0619 16:49:59.865163 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.865190 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.865240 (+ 50us) server_negotiation.cc:407] Connection header received
0619 16:49:59.866066 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.866074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.866434 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.866695 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.868045 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.869006 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.870252 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.870704 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.870873 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.878892 (+ 8019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.878913 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.878963 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:59.879009 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.881073 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:59.881787 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:59.881794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:59.881800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:59.881981 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:59.882394 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:59.882400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:59.882403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:59.882888 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.883094 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.883623 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.883893 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":125,"threads_started":1}
I20260619 16:49:59.884138 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.864683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:41825 (local address 127.3.233.129:57173)
0619 16:49:59.864937 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.864953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.865064 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.866809 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.866815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.867392 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:59.867871 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.867887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.869168 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.869171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.870104 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.870113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.870837 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.870840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.871108 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.876614 (+ 5506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:59.876639 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:59.878508 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:59.881231 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:59.881238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:59.881253 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:59.881614 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:59.882085 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:59.882088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:59.882090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:59.882300 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:59.883066 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:59.883073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:59.883306 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.883498 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.883970 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":316,"spinlock_wait_cycles":28928,"thread_start_us":100,"threads_started":1}
I20260619 16:49:59.884115 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.865393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:46321 (local address 127.3.233.130:44721)
0619 16:49:59.865697 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.865707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.865737 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:49:59.866356 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.866361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.866715 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.867027 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.868051 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.868998 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.870218 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.870702 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.870879 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.879156 (+ 8277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.879161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.879172 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:59.879205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.881243 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:59.881958 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:59.881963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:59.881967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:59.882145 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:59.882456 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:59.882459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:59.882460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:59.882886 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.883108 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.883612 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.883907 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
W20260619 16:49:59.887149 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
W20260619 16:49:59.887385 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
I20260619 16:49:59.887457 20563 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e; no voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6
I20260619 16:49:59.887766 20948 raft_consensus.cc:2726] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:50:00.107148 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.106717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:55985)
0619 16:50:00.106899 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:00.107055 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 16:50:00.275220 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.274726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:40105)
0619 16:50:00.275049 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:00.275112 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":73,"threads_started":1}
I20260619 16:50:00.472564 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.472108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:46791)
0619 16:50:00.472379 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:00.472433 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260619 16:50:01.108413 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.107937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:59283)
0619 16:50:01.108224 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.108305 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260619 16:50:01.276335 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.275947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:44859)
0619 16:50:01.276188 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.276243 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260619 16:50:01.473798 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.473383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:56617)
0619 16:50:01.473628 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.473691 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260619 16:50:02.103283 20958 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:02.103410 20958 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:50:02.103914 20958 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
W20260619 16:50:02.104779 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
W20260619 16:50:02.104911 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
I20260619 16:50:02.104970 20563 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e; no voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6
I20260619 16:50:02.105163 20958 raft_consensus.cc:2726] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:50:02.109588 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:02.109123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:33921)
0619 16:50:02.109431 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:02.109488 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":111,"threads_started":1}
I20260619 16:50:02.277709 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:02.277132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:59587)
0619 16:50:02.277455 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:02.277553 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260619 16:50:02.475241 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:02.474694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:59899)
0619 16:50:02.475044 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:02.475107 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260619 16:50:03.111002 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:03.110507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:34123)
0619 16:50:03.110816 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:03.110878 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":91,"threads_started":1}
I20260619 16:50:03.278896 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:03.278505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:54037)
0619 16:50:03.278745 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:03.278803 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260619 16:50:03.476473 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:03.476051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:49241)
0619 16:50:03.476315 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:03.476379 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260619 16:50:03.877696 20965 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:03.877847 20965 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:50:03.878346 20965 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
W20260619 16:50:03.879274 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
W20260619 16:50:03.879473 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
I20260619 16:50:03.879560 20563 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e; no voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6
I20260619 16:50:03.879715 20965 raft_consensus.cc:2726] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:50:04.112365 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:04.111825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:45119)
0619 16:50:04.112178 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:04.112246 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":92,"threads_started":1}
I20260619 16:50:04.280208 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:04.279742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:35127)
0619 16:50:04.280039 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:04.280107 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
I20260619 16:50:04.477854 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:04.477364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:56947)
0619 16:50:04.477659 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:04.477751 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260619 16:50:05.113886 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:05.113385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:53519)
0619 16:50:05.113709 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:05.113780 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260619 16:50:05.281730 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:05.281190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:52123)
0619 16:50:05.281557 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:05.281631 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260619 16:50:05.479326 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:05.478752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:50273)
0619 16:50:05.479153 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:05.479226 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":74,"threads_started":1}
I20260619 16:50:06.115401 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.114838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:39999)
0619 16:50:06.115239 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:06.115302 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
I20260619 16:50:06.283088 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.282606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:55503)
0619 16:50:06.282855 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:06.282909 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260619 16:50:06.289619 20974 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:06.289763 20974 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:50:06.290148 20974 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
W20260619 16:50:06.291046 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
W20260619 16:50:06.291224 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
I20260619 16:50:06.291285 20563 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e; no voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6
I20260619 16:50:06.291479 20974 raft_consensus.cc:2726] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:50:06.480768 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:06.480251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:60405)
0619 16:50:06.480557 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:06.480618 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":122,"threads_started":1}
I20260619 16:50:07.116596 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:07.116192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:36163)
0619 16:50:07.116442 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:07.116502 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260619 16:50:07.284545 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:07.283985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:42005)
0619 16:50:07.284373 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:07.284442 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":166,"threads_started":1}
I20260619 16:50:07.482147 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:07.481688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:46307)
0619 16:50:07.481986 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:07.482054 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
I20260619 16:50:08.117928 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:08.117423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:56795)
0619 16:50:08.117772 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:08.117832 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260619 16:50:08.285866 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:08.285452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:35823)
0619 16:50:08.285708 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:08.285769 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260619 16:50:08.483448 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:08.483050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:34239)
0619 16:50:08.483298 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:08.483355 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260619 16:50:09.119457 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:09.118856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:33339)
0619 16:50:09.119255 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:09.119351 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":179,"threads_started":1}
I20260619 16:50:09.287324 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:09.286773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:42649)
0619 16:50:09.287149 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:09.287220 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":81,"threads_started":1}
I20260619 16:50:09.484818 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:09.484357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:40513)
0619 16:50:09.484634 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:09.484721 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260619 16:50:09.553529 20985 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:09.553709 20985 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:50:09.554169 20985 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
W20260619 16:50:09.555188 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
W20260619 16:50:09.555372 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
I20260619 16:50:09.555441 20563 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e; no voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6
I20260619 16:50:09.555645 20985 raft_consensus.cc:2726] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:50:10.120862 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:10.120398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:51253)
0619 16:50:10.120680 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:10.120762 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260619 16:50:10.288692 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:10.288224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:41513)
0619 16:50:10.288514 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:10.288583 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
I20260619 16:50:10.486114 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:10.485637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:60143)
0619 16:50:10.485938 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:10.486003 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
I20260619 16:50:11.122309 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:11.121782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:53521)
0619 16:50:11.122083 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:11.122166 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
I20260619 16:50:11.289872 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:11.289467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:56141)
0619 16:50:11.289717 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:11.289774 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260619 16:50:11.487629 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:11.487141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:53105)
0619 16:50:11.487455 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:11.487525 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260619 16:50:11.488786 20992 raft_consensus.cc:482] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:11.488935 20992 raft_consensus.cc:504] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f24891cac5e495084b9a02bbec846af" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 41825 } } peers { permanent_uuid: "f851bee8884e44d490d4de3756fa86f6" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 44721 } } peers { permanent_uuid: "ebc41f368a23471889f7c3abf244f74e" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 39145 } }
I20260619 16:50:11.489339 20992 leader_election.cc:288] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825), f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721)
W20260619 16:50:11.490082 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f24891cac5e495084b9a02bbec846af (127.3.233.131:41825): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
W20260619 16:50:11.490242 20563 leader_election.cc:341] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f851bee8884e44d490d4de3756fa86f6 (127.3.233.130:44721): Not found: Tablet not found: 35e2825c846049a69b83fe5890c03e82
I20260619 16:50:11.490286 20563 leader_election.cc:302] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc41f368a23471889f7c3abf244f74e; no voters: 9f24891cac5e495084b9a02bbec846af, f851bee8884e44d490d4de3756fa86f6
I20260619 16:50:11.490413 20992 raft_consensus.cc:2726] T 35e2825c846049a69b83fe5890c03e82 P ebc41f368a23471889f7c3abf244f74e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:50:12.123790 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:12.123253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:38863)
0619 16:50:12.123629 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:12.123693 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":103,"threads_started":1}
I20260619 16:50:12.291370 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:12.290765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:35751)
0619 16:50:12.291193 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:12.291270 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":82,"threads_started":1}
I20260619 16:50:12.489209 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:12.488648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:51351)
0619 16:50:12.488972 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:12.489056 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260619 16:50:13.125393 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:13.124960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.129:53313)
0619 16:50:13.125236 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:13.125300 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
I20260619 16:50:13.292732 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:13.292279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.130:49511)
0619 16:50:13.292566 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:13.292638 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260619 16:50:13.480382 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 20505
I20260619 16:50:13.486090 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 20678
I20260619 16:50:13.490747 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:13.490193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32853 (local address 127.3.233.131:42057)
0619 16:50:13.490509 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:13.490590 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.190:32853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
I20260619 16:50:13.492084 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 20812
2026-06-19T16:50:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59429 ms)
[----------] 30 tests from SecurityITest (243972 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:13 dist-test-slave-cgwx krb5kdc[21001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:13 dist-test-slave-cgwx krb5kdc[21001](info): set up 2 sockets
Jun 19 16:50:13 dist-test-slave-cgwx krb5kdc[21001](info): commencing operation
krb5kdc: starting...
W20260619 16:50:14.658784 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:14 dist-test-slave-cgwx krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:14.780232 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32977
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:32977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:14.892359 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:14.892640 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:14.892720 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:14.896968 21014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:14.897030 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:14.897050 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:14.897068 21014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:14.897116 21014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:14.902455 21014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:32977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:32977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:14.903592 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:14.904628 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:14.913316 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:14.913316 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:14.913775 21014 server_base.cc:1034] running on GCE node
W20260619 16:50:14.913316 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:14.914494 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:14.915777 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:14.917014 21014 hybrid_clock.cc:648] HybridClock initialized: now 1781887814916980 us; error 50 us; skew 500 ppm
Jun 19 16:50:14 dist-test-slave-cgwx krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:14.920208 21014 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:14.921231 21014 webserver.cc:460] Webserver started at http://127.3.233.190:36791/ using document root <none> and password file <none>
I20260619 16:50:14.921823 21014 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:14.921916 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:14.922160 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:14.924280 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "a46b6ae724c2440da7dbf3f2b32f8946"
format_stamp: "Formatted at 2026-06-19 16:50:14 on dist-test-slave-cgwx"
server_key: "a930001b0b8e05c630935beffc412617"
server_key_iv: "d46d3f01b58f7a3500c3ff49671cc7f5"
server_key_version: "encryptionkey@0"
I20260619 16:50:14.924841 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "a46b6ae724c2440da7dbf3f2b32f8946"
format_stamp: "Formatted at 2026-06-19 16:50:14 on dist-test-slave-cgwx"
server_key: "a930001b0b8e05c630935beffc412617"
server_key_iv: "d46d3f01b58f7a3500c3ff49671cc7f5"
server_key_version: "encryptionkey@0"
I20260619 16:50:14.928802 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 16:50:14.931226 21029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:14.932622 21014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:50:14.932782 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "a46b6ae724c2440da7dbf3f2b32f8946"
format_stamp: "Formatted at 2026-06-19 16:50:14 on dist-test-slave-cgwx"
server_key: "a930001b0b8e05c630935beffc412617"
server_key_iv: "d46d3f01b58f7a3500c3ff49671cc7f5"
server_key_version: "encryptionkey@0"
I20260619 16:50:14.932905 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:14.951349 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:14.955809 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:14.956069 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:14.964728 21014 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:32977
I20260619 16:50:14.964753 21081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:32977 every 8 connection(s)
I20260619 16:50:14.965732 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:14.967000 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21014
I20260619 16:50:14.967140 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:14.967424 4006 external_mini_cluster.cc:1428] Setting key 831a2a3121a42fec1ab971c5d66b0c3d
I20260619 16:50:14.968928 21082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:14.975956 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946: Bootstrap starting.
Jun 19 16:50:14 dist-test-slave-cgwx krb5kdc[21001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:14.978711 21082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:14.979525 21082 log.cc:824] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:14.981398 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946: No bootstrap required, opened a new log
I20260619 16:50:14.982503 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:14.968847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.3.233.190:32977)
0619 16:50:14.969262 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:14.969281 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:14.969334 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:50:14.970151 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:14.970173 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:14.970597 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:14.970977 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:14.971677 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.973180 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.974022 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:14.974747 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:14.974918 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:14.977831 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:14.977837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:14.977854 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:14.977885 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:14.979810 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:14.980252 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:14.980257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:14.980261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:14.980368 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:14.980656 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:14.980660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:14.980664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:14.981043 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:14.981247 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:14.981534 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 16:50:14.981830 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
I20260619 16:50:14.985080 21082 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32977 } }
I20260619 16:50:14.985328 21082 raft_consensus.cc:374] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:14.985411 21082 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46b6ae724c2440da7dbf3f2b32f8946, State: Initialized, Role: FOLLOWER
I20260619 16:50:14.985841 21082 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32977 } }
I20260619 16:50:14.986002 21082 raft_consensus.cc:388] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:14.986105 21082 raft_consensus.cc:482] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:14.986223 21082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:14.987216 21082 raft_consensus.cc:504] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32977 } }
I20260619 16:50:14.987558 21082 leader_election.cc:302] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a46b6ae724c2440da7dbf3f2b32f8946; no voters:
I20260619 16:50:14.987895 21082 leader_election.cc:288] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:14.988032 21087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:14.988346 21087 raft_consensus.cc:686] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [term 1 LEADER]: Becoming Leader. State: Replica: a46b6ae724c2440da7dbf3f2b32f8946, State: Running, Role: LEADER
I20260619 16:50:14.988633 21087 consensus_queue.cc:237] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32977 } }
I20260619 16:50:14.989153 21082 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:14.991348 21089 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a46b6ae724c2440da7dbf3f2b32f8946. Latest consensus state: current_term: 1 leader_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32977 } } }
I20260619 16:50:14.991488 21089 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:14.991763 21088 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a46b6ae724c2440da7dbf3f2b32f8946" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 32977 } } }
I20260619 16:50:14.991852 21088 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:14.992147 21097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:14.994818 21097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:15.000710 21097 catalog_manager.cc:1347] Generated new cluster ID: 87c42b85e6684999bd72319b3d3701bb
I20260619 16:50:15.000797 21097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:15.011088 21097 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:15.012091 21097 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:15.023079 21097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946: Generated new TSK 0
I20260619 16:50:15.023850 21097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:50:15.058399 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32977
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:15.172034 21108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:15.172317 21108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:15.172360 21108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:15.176580 21108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:15.176649 21108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:15.176712 21108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:50:15.182134 21108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:15.183378 21108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:15.184494 21108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:15.194787 21113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:15.194802 21116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:15.194802 21114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:15.195057 21108 server_base.cc:1034] running on GCE node
I20260619 16:50:15.195585 21108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:15.196269 21108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:15.197490 21108 hybrid_clock.cc:648] HybridClock initialized: now 1781887815197450 us; error 52 us; skew 500 ppm
Jun 19 16:50:15 dist-test-slave-cgwx krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:15.200573 21108 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:50:15.201498 21108 webserver.cc:460] Webserver started at http://127.3.233.129:45551/ using document root <none> and password file <none>
I20260619 16:50:15.202098 21108 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:15.202193 21108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:15.202433 21108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:15.204712 21108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "a3e09cb1a5804dd58a2aa15f3b794358"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "d430d1cd99aedd07f1fc3d7589549ae4"
server_key_iv: "9d1e289eff13bc73e2dd977723917f4b"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.205310 21108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "a3e09cb1a5804dd58a2aa15f3b794358"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "d430d1cd99aedd07f1fc3d7589549ae4"
server_key_iv: "9d1e289eff13bc73e2dd977723917f4b"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.209400 21108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:15.211980 21123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.213238 21108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:50:15.213375 21108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "a3e09cb1a5804dd58a2aa15f3b794358"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "d430d1cd99aedd07f1fc3d7589549ae4"
server_key_iv: "9d1e289eff13bc73e2dd977723917f4b"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.213492 21108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:15.223196 21108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:15.227103 21108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:15.227329 21108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:15.228036 21108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:15.228977 21108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:15.229053 21108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.229130 21108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:15.229183 21108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.238842 21108 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:34159
I20260619 16:50:15.238881 21236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:34159 every 8 connection(s)
I20260619 16:50:15.239917 21108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:15.245253 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21108
I20260619 16:50:15.245389 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:15.245709 4006 external_mini_cluster.cc:1428] Setting key fe1afbe7b384f72ddbd6175fa37eb0ce
Jun 19 16:50:15 dist-test-slave-cgwx krb5kdc[21001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:50:15.256367 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.241928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:51417 (local address 127.3.233.190:32977)
0619 16:50:15.242067 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.242071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.242934 (+ 863us) server_negotiation.cc:407] Connection header received
0619 16:50:15.244312 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.244316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.244374 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.244454 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:15.246103 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.246877 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.248826 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.249085 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.249235 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.252026 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:15.252033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:15.252040 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:15.252078 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:15.253942 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.254496 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.254502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.254503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.254601 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.255043 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.255046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.255048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.255326 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:15.255444 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:15.255989 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 16:50:15.256156 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:50:15.257221 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.242294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32977 (local address 127.3.233.129:51417)
0619 16:50:15.242760 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.242804 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.244063 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.244641 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.244648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.245137 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:15.245885 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.245908 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.247043 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.247046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.248655 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.248668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.249281 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.249285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.249484 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.250145 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:15.250163 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:15.251847 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:15.254072 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:15.254077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:15.254091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:15.254380 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:15.254710 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:15.254712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:15.254714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:15.254881 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:15.255441 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:15.255447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:15.255702 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 16:50:15.255883 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:50:15.256246 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:50:15.258628 21237 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32977
I20260619 16:50:15.258956 21237 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:15.259553 21237 heartbeater.cc:511] Master 127.3.233.190:32977 requested a full tablet report, sending...
I20260619 16:50:15.261327 21046 ts_manager.cc:194] Registered new tserver with Master: a3e09cb1a5804dd58a2aa15f3b794358 (127.3.233.129:34159)
I20260619 16:50:15.262501 21046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:51417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:50:15.278429 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32977
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:15.389568 21242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:15.389864 21242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:15.389909 21242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:15.394327 21242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:15.394389 21242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:15.394483 21242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:50:15.399755 21242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:15.400991 21242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:15.402081 21242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:15.411788 21250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:15.411777 21247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:15.411893 21242 server_base.cc:1034] running on GCE node
W20260619 16:50:15.411772 21248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:15.412536 21242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:15.413266 21242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:15.414448 21242 hybrid_clock.cc:648] HybridClock initialized: now 1781887815414424 us; error 36 us; skew 500 ppm
Jun 19 16:50:15 dist-test-slave-cgwx krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:15.417794 21242 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:50:15.418809 21242 webserver.cc:460] Webserver started at http://127.3.233.130:43805/ using document root <none> and password file <none>
I20260619 16:50:15.419488 21242 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:15.419590 21242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:15.419868 21242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:15.422012 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "047b0407260f49dd99d40d89cd59fcad"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "d1676daf254c8abbc7e1db2d401d6939"
server_key_iv: "1f910292bb02b3225eec11d201e68fed"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.422598 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "047b0407260f49dd99d40d89cd59fcad"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "d1676daf254c8abbc7e1db2d401d6939"
server_key_iv: "1f910292bb02b3225eec11d201e68fed"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.426625 21242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:50:15.429343 21257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.430712 21242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:15.430892 21242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "047b0407260f49dd99d40d89cd59fcad"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "d1676daf254c8abbc7e1db2d401d6939"
server_key_iv: "1f910292bb02b3225eec11d201e68fed"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.431068 21242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:15.465373 21242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:15.468806 21242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:15.469024 21242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:15.469626 21242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:15.470458 21242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:15.470530 21242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.470598 21242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:15.470649 21242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.480050 21242 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:36737
I20260619 16:50:15.480073 21370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:36737 every 8 connection(s)
I20260619 16:50:15.480993 21242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:15.485489 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21242
I20260619 16:50:15.485626 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:15.485982 4006 external_mini_cluster.cc:1428] Setting key fb4d47850f66a091edcbf1076a374313
Jun 19 16:50:15 dist-test-slave-cgwx krb5kdc[21001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:50:15.498375 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.483001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:59595 (local address 127.3.233.190:32977)
0619 16:50:15.483143 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.483147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.484355 (+ 1208us) server_negotiation.cc:407] Connection header received
0619 16:50:15.486326 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.486330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.486394 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.486453 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:15.488140 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.489025 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.490909 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.491192 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.491368 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.494508 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:15.494515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:15.494517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:15.494549 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:15.496340 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.496890 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.496896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.496898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.496998 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.497365 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.497367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.497369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.497556 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:15.497636 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:15.498137 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 16:50:15.498246 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:50:15.499384 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.483520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32977 (local address 127.3.233.130:59595)
0619 16:50:15.484156 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.484216 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.485984 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.486893 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.486900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.487445 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:15.487987 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.488000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.489209 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.489214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.490787 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.490800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.491413 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.491417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.491596 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.492343 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:15.492371 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:15.494312 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:15.496482 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:15.496487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:15.496501 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:15.496780 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:15.497105 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:15.497107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:15.497109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:15.497269 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:15.497657 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:15.497663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:15.497915 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 16:50:15.498079 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 16:50:15.498455 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":140,"threads_started":1}
I20260619 16:50:15.500726 21371 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32977
I20260619 16:50:15.501004 21371 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:15.501705 21371 heartbeater.cc:511] Master 127.3.233.190:32977 requested a full tablet report, sending...
I20260619 16:50:15.503012 21046 ts_manager.cc:194] Registered new tserver with Master: 047b0407260f49dd99d40d89cd59fcad (127.3.233.130:36737)
I20260619 16:50:15.503693 21046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:59595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:50:15.518508 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32977
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:15.632200 21376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:15.632542 21376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:15.632619 21376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:15.637077 21376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:15.637197 21376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:15.637296 21376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:50:15.642771 21376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:32977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:15.643973 21376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:15.645115 21376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:15.655525 21382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:15.655685 21381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:15.655508 21384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:15.655725 21376 server_base.cc:1034] running on GCE node
I20260619 16:50:15.656472 21376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:15.657186 21376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:15.658432 21376 hybrid_clock.cc:648] HybridClock initialized: now 1781887815658415 us; error 29 us; skew 500 ppm
Jun 19 16:50:15 dist-test-slave-cgwx krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:15.661680 21376 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:50:15.662740 21376 webserver.cc:460] Webserver started at http://127.3.233.131:33227/ using document root <none> and password file <none>
I20260619 16:50:15.663411 21376 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:15.663481 21376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:15.663667 21376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:15.665874 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "c9a11e4e892d4269bb945aed2a3763ce"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "e0a918520c0ab96568c713d6cd95e4fe"
server_key_iv: "8c562b9b81e002ebd3ebc6e9ee74c013"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.666471 21376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "c9a11e4e892d4269bb945aed2a3763ce"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "e0a918520c0ab96568c713d6cd95e4fe"
server_key_iv: "8c562b9b81e002ebd3ebc6e9ee74c013"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.670562 21376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260619 16:50:15.673362 21391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.674788 21376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:15.674948 21376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "c9a11e4e892d4269bb945aed2a3763ce"
format_stamp: "Formatted at 2026-06-19 16:50:15 on dist-test-slave-cgwx"
server_key: "e0a918520c0ab96568c713d6cd95e4fe"
server_key_iv: "8c562b9b81e002ebd3ebc6e9ee74c013"
server_key_version: "encryptionkey@0"
I20260619 16:50:15.675155 21376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:15.693547 21376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:15.697114 21376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:15.697351 21376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:15.697993 21376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:15.698837 21376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:15.698884 21376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.698985 21376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:15.699028 21376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:15.708767 21376 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:39541
I20260619 16:50:15.708792 21504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:39541 every 8 connection(s)
I20260619 16:50:15.709725 21376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:15.715234 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21376
I20260619 16:50:15.715410 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:15.715730 4006 external_mini_cluster.cc:1428] Setting key ca8332782620934f42ed39fce7bfced4
Jun 19 16:50:15 dist-test-slave-cgwx krb5kdc[21001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:15.724807 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.711723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:57995 (local address 127.3.233.190:32977)
0619 16:50:15.711873 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.711877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.712762 (+ 885us) server_negotiation.cc:407] Connection header received
0619 16:50:15.713992 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.713995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.714043 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.714118 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:15.715577 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.716276 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.718066 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.718338 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.718427 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.721109 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:15.721112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:15.721114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:15.721140 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:15.722648 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.723252 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.723255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.723256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.723331 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.723736 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.723738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.723739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.723928 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:15.724001 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:15.724552 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 16:50:15.724663 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:50:15.725657 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.712124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:32977 (local address 127.3.233.131:57995)
0619 16:50:15.712584 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.712637 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.713790 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.714315 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.714322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.714691 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:15.715405 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.715428 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.716477 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.716480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.717939 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.717949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.718467 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.718470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.718606 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.719280 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:15.719296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:15.720938 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:15.722798 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:15.722802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:15.722816 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:15.723154 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:15.723442 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:15.723445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:15.723446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:15.723622 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:15.724023 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:15.724028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:15.724308 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 16:50:15.724478 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:50:15.724772 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260619 16:50:15.727097 21505 heartbeater.cc:347] Connected to a master server at 127.3.233.190:32977
I20260619 16:50:15.727383 21505 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:15.727883 21505 heartbeater.cc:511] Master 127.3.233.190:32977 requested a full tablet report, sending...
I20260619 16:50:15.729176 21045 ts_manager.cc:194] Registered new tserver with Master: c9a11e4e892d4269bb945aed2a3763ce (127.3.233.131:39541)
I20260619 16:50:15.730043 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:15.730144 21045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:57995
I20260619 16:50:15.740832 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.732399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.3.233.190:32977)
0619 16:50:15.732528 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.732532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.732677 (+ 145us) server_negotiation.cc:407] Connection header received
0619 16:50:15.732766 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.732768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.732805 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.732878 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:15.733588 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.734184 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.735293 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.735532 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.735621 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.737444 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:15.737448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:15.737450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:15.737475 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:15.739165 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.739604 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.739607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.739608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.739689 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:15.739973 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:15.739975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:15.739976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:15.740192 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:15.740300 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:15.740601 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:50:15.740701 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:50:15.744076 21045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:15.747226 21045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:15.763068 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.757134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34159 (local address 127.0.0.1:49328)
0619 16:50:15.758145 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.758159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.758247 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.758827 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.758830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.758860 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:15.759065 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.759074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.760374 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.760378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.761575 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.761585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.762473 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.762476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.762588 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.762624 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:50:15.762768 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:50:15.762892 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":249,"spinlock_wait_cycles":8320,"thread_start_us":95,"threads_started":1}
I20260619 16:50:15.763079 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.757528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.3.233.129:34159)
0619 16:50:15.758089 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.758108 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.758281 (+ 173us) server_negotiation.cc:407] Connection header received
0619 16:50:15.758349 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.758354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.758525 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.758672 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:15.759190 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.760288 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.761726 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.762347 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.762473 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.762626 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:15.762810 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:50:15.762903 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":100,"threads_started":1}
I20260619 16:50:15.763756 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.757372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.3.233.131:39541)
0619 16:50:15.757866 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.757880 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.758244 (+ 364us) server_negotiation.cc:407] Connection header received
0619 16:50:15.758332 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.758337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.758504 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.758661 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:15.759016 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.760203 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.761880 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.762687 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.762800 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.762969 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:15.763515 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 16:50:15.763607 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":139,"threads_started":1}
I20260619 16:50:15.763422 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.756866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:39541 (local address 127.0.0.1:56788)
0619 16:50:15.758066 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.758081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.758172 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.758667 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.758673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.758694 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:15.758847 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.758853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.760420 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.760423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.761764 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.761773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.763028 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.763033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.763127 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.763149 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:50:15.763220 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:50:15.763285 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":2,"thread_start_us":110,"threads_started":1}
I20260619 16:50:15.764530 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.756685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:36737 (local address 127.0.0.1:60012)
0619 16:50:15.757530 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:15.757582 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:15.757757 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:15.758827 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:15.758830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:15.758854 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:15.759076 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.759082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.761658 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.761662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.762813 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:15.762823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.764111 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.764117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:15.764228 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.764254 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:50:15.764325 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:50:15.764374 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":50,"spinlock_wait_cycles":9728,"thread_start_us":87,"threads_started":1}
I20260619 16:50:15.764643 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:15.756867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.3.233.130:36737)
0619 16:50:15.758183 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:15.758196 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:15.758235 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:50:15.758318 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:15.758323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:15.758525 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:15.758673 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:15.760389 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.761513 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.762966 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:15.763817 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:15.763921 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:15.764047 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:15.764383 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:50:15.764488 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214,"thread_start_us":106,"threads_started":1}
I20260619 16:50:15.766902 21171 tablet_service.cc:1467] Processing CreateTablet for tablet 08386ea4082b47f1a0fa9b51836f4cbf (DEFAULT_TABLE table=test-table [id=dda68fc0edd148418a52d2d3061b992a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:15.767163 21439 tablet_service.cc:1467] Processing CreateTablet for tablet 08386ea4082b47f1a0fa9b51836f4cbf (DEFAULT_TABLE table=test-table [id=dda68fc0edd148418a52d2d3061b992a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:15.767514 21305 tablet_service.cc:1467] Processing CreateTablet for tablet 08386ea4082b47f1a0fa9b51836f4cbf (DEFAULT_TABLE table=test-table [id=dda68fc0edd148418a52d2d3061b992a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:15.768205 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08386ea4082b47f1a0fa9b51836f4cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:15.768246 21439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08386ea4082b47f1a0fa9b51836f4cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:15.768393 21305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08386ea4082b47f1a0fa9b51836f4cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:15.774761 21522 tablet_bootstrap.cc:492] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358: Bootstrap starting.
I20260619 16:50:15.775511 21523 tablet_bootstrap.cc:492] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Bootstrap starting.
I20260619 16:50:15.776691 21524 tablet_bootstrap.cc:492] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Bootstrap starting.
I20260619 16:50:15.777421 21522 tablet_bootstrap.cc:654] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:15.777809 21523 tablet_bootstrap.cc:654] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:15.778505 21522 log.cc:824] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:15.779104 21524 tablet_bootstrap.cc:654] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:15.779229 21523 log.cc:824] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:15.780079 21524 log.cc:824] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:15.780869 21522 tablet_bootstrap.cc:492] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358: No bootstrap required, opened a new log
I20260619 16:50:15.781097 21522 ts_tablet_manager.cc:1397] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:50:15.781445 21523 tablet_bootstrap.cc:492] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: No bootstrap required, opened a new log
I20260619 16:50:15.781679 21523 ts_tablet_manager.cc:1397] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:50:15.782089 21524 tablet_bootstrap.cc:492] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: No bootstrap required, opened a new log
I20260619 16:50:15.782266 21524 ts_tablet_manager.cc:1397] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 16:50:15.784898 21523 raft_consensus.cc:348] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:15.785169 21523 raft_consensus.cc:374] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:15.785247 21523 raft_consensus.cc:729] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a11e4e892d4269bb945aed2a3763ce, State: Initialized, Role: FOLLOWER
I20260619 16:50:15.785734 21523 consensus_queue.cc:260] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:15.786118 21524 raft_consensus.cc:348] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:15.786386 21524 raft_consensus.cc:374] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:15.786445 21524 raft_consensus.cc:729] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047b0407260f49dd99d40d89cd59fcad, State: Initialized, Role: FOLLOWER
I20260619 16:50:15.786826 21523 ts_tablet_manager.cc:1428] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 16:50:15.786687 21522 raft_consensus.cc:348] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:15.786847 21524 consensus_queue.cc:260] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:15.786964 21522 raft_consensus.cc:374] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:15.787016 21522 raft_consensus.cc:729] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e09cb1a5804dd58a2aa15f3b794358, State: Initialized, Role: FOLLOWER
I20260619 16:50:15.787257 21505 heartbeater.cc:503] Master 127.3.233.190:32977 was elected leader, sending a full tablet report...
I20260619 16:50:15.787470 21522 consensus_queue.cc:260] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:15.787823 21524 ts_tablet_manager.cc:1428] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 16:50:15.788075 21237 heartbeater.cc:503] Master 127.3.233.190:32977 was elected leader, sending a full tablet report...
I20260619 16:50:15.788118 21371 heartbeater.cc:503] Master 127.3.233.190:32977 was elected leader, sending a full tablet report...
I20260619 16:50:15.788407 21522 ts_tablet_manager.cc:1428] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
W20260619 16:50:15.960704 21506 tablet.cc:2369] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:50:15.982184 21372 tablet.cc:2369] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:50:15.991569 21238 tablet.cc:2369] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:16.106861 21528 raft_consensus.cc:482] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:16.107144 21528 raft_consensus.cc:504] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:16.108282 21528 leader_election.cc:288] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047b0407260f49dd99d40d89cd59fcad (127.3.233.130:36737), a3e09cb1a5804dd58a2aa15f3b794358 (127.3.233.129:34159)
I20260619 16:50:16.112610 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.108588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:36737 (local address 127.3.233.131:53557)
0619 16:50:16.108710 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.108727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:16.108827 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:16.109130 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:16.109133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:16.109150 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:16.109356 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.109367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.110288 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.110291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.111408 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.111419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.112102 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.112106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.112228 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.112312 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 16:50:16.112381 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:50:16.112443 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":23}
I20260619 16:50:16.112649 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.108671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:53557 (local address 127.3.233.130:36737)
0619 16:50:16.108832 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.108835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.108850 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:50:16.108958 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.108961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.109016 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.109109 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:16.109509 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.110166 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.111538 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.111980 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.112080 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.112122 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:16.112425 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 16:50:16.112516 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:50:16.112789 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.108588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34159 (local address 127.3.233.131:39933)
0619 16:50:16.109235 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.109247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:16.109334 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:16.109571 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:16.109573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:16.109591 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:16.109724 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.109731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.110601 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.110604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.111489 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.111503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.112235 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.112238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.112423 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.112448 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:50:16.112522 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:50:16.112586 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":342,"thread_start_us":136,"threads_started":1}
I20260619 16:50:16.113262 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.108870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:39933 (local address 127.3.233.129:34159)
0619 16:50:16.109015 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.109021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.109331 (+ 310us) server_negotiation.cc:407] Connection header received
0619 16:50:16.109410 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.109416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.109474 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.109550 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:16.109871 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.110485 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.111617 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.112096 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.112225 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.112285 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:16.112555 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 16:50:16.112635 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:50:16.113384 21325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08386ea4082b47f1a0fa9b51836f4cbf" candidate_uuid: "c9a11e4e892d4269bb945aed2a3763ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047b0407260f49dd99d40d89cd59fcad" is_pre_election: true
I20260619 16:50:16.113735 21325 raft_consensus.cc:2454] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a11e4e892d4269bb945aed2a3763ce in term 0.
I20260619 16:50:16.113790 21191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08386ea4082b47f1a0fa9b51836f4cbf" candidate_uuid: "c9a11e4e892d4269bb945aed2a3763ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" is_pre_election: true
I20260619 16:50:16.114141 21191 raft_consensus.cc:2454] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a11e4e892d4269bb945aed2a3763ce in term 0.
I20260619 16:50:16.114307 21395 leader_election.cc:302] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047b0407260f49dd99d40d89cd59fcad, c9a11e4e892d4269bb945aed2a3763ce; no voters:
I20260619 16:50:16.114617 21528 raft_consensus.cc:2781] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:16.114725 21528 raft_consensus.cc:482] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:16.114806 21528 raft_consensus.cc:3037] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:16.115931 21528 raft_consensus.cc:504] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:16.116338 21528 leader_election.cc:288] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [CANDIDATE]: Term 1 election: Requested vote from peers 047b0407260f49dd99d40d89cd59fcad (127.3.233.130:36737), a3e09cb1a5804dd58a2aa15f3b794358 (127.3.233.129:34159)
I20260619 16:50:16.116693 21191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08386ea4082b47f1a0fa9b51836f4cbf" candidate_uuid: "c9a11e4e892d4269bb945aed2a3763ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e09cb1a5804dd58a2aa15f3b794358"
I20260619 16:50:16.116758 21325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08386ea4082b47f1a0fa9b51836f4cbf" candidate_uuid: "c9a11e4e892d4269bb945aed2a3763ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047b0407260f49dd99d40d89cd59fcad"
I20260619 16:50:16.116803 21191 raft_consensus.cc:3037] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:16.116914 21325 raft_consensus.cc:3037] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:16.118218 21191 raft_consensus.cc:2454] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a11e4e892d4269bb945aed2a3763ce in term 1.
I20260619 16:50:16.118374 21325 raft_consensus.cc:2454] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a11e4e892d4269bb945aed2a3763ce in term 1.
I20260619 16:50:16.118636 21394 leader_election.cc:302] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e09cb1a5804dd58a2aa15f3b794358, c9a11e4e892d4269bb945aed2a3763ce; no voters:
I20260619 16:50:16.118876 21528 raft_consensus.cc:2781] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:16.119150 21528 raft_consensus.cc:686] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 1 LEADER]: Becoming Leader. State: Replica: c9a11e4e892d4269bb945aed2a3763ce, State: Running, Role: LEADER
I20260619 16:50:16.119494 21528 consensus_queue.cc:237] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } }
I20260619 16:50:16.122840 21044 catalog_manager.cc:5495] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a11e4e892d4269bb945aed2a3763ce (127.3.233.131). New cstate: current_term: 1 leader_uuid: "c9a11e4e892d4269bb945aed2a3763ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a11e4e892d4269bb945aed2a3763ce" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 39541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 } health_report { overall_health: UNKNOWN } } }
I20260619 16:50:16.215690 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.212046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.3.233.131:39541)
0619 16:50:16.212174 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.212178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.212193 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:50:16.212345 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.212348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.212400 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.212496 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:16.212718 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.213228 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.214151 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.214372 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.214452 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.214722 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:16.215356 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:16.215452 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:16.215524 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:50:16.215576 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:50:16.221684 21191 raft_consensus.cc:1261] T 08386ea4082b47f1a0fa9b51836f4cbf P a3e09cb1a5804dd58a2aa15f3b794358 [term 1 FOLLOWER]: Refusing update from remote peer c9a11e4e892d4269bb945aed2a3763ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:16.221678 21325 raft_consensus.cc:1261] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 1 FOLLOWER]: Refusing update from remote peer c9a11e4e892d4269bb945aed2a3763ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:16.222492 21528 consensus_queue.cc:1035] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e09cb1a5804dd58a2aa15f3b794358" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:16.222678 21532 consensus_queue.cc:1035] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "047b0407260f49dd99d40d89cd59fcad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:16.241885 21046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:16.242089 21046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:16.244642 21046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a46b6ae724c2440da7dbf3f2b32f8946: Sending DeleteTablet for 3 replicas of tablet 08386ea4082b47f1a0fa9b51836f4cbf
I20260619 16:50:16.245625 21171 tablet_service.cc:1514] Processing DeleteTablet for tablet 08386ea4082b47f1a0fa9b51836f4cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:49328
I20260619 16:50:16.245625 21305 tablet_service.cc:1514] Processing DeleteTablet for tablet 08386ea4082b47f1a0fa9b51836f4cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:60012
I20260619 16:50:16.246464 21439 tablet_service.cc:1514] Processing DeleteTablet for tablet 08386ea4082b47f1a0fa9b51836f4cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:16 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:56788
I20260619 16:50:16.247236 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21108
I20260619 16:50:16.247591 21545 tablet_replica.cc:330] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: stopping tablet replica
I20260619 16:50:16.247937 21545 raft_consensus.cc:2229] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:16.248301 21545 raft_consensus.cc:2258] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:16.249711 21545 ts_tablet_manager.cc:1905] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:16.249874 21547 tablet_replica.cc:330] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: stopping tablet replica
I20260619 16:50:16.250180 21547 raft_consensus.cc:2229] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:16.250586 21547 raft_consensus.cc:2258] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:16.252087 21547 ts_tablet_manager.cc:1905] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:16.252524 21545 ts_tablet_manager.cc:1918] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:16.252626 21545 log.cc:1196] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/08386ea4082b47f1a0fa9b51836f4cbf
I20260619 16:50:16.252959 21545 ts_tablet_manager.cc:1939] T 08386ea4082b47f1a0fa9b51836f4cbf P 047b0407260f49dd99d40d89cd59fcad: Deleting consensus metadata
I20260619 16:50:16.254285 21033 catalog_manager.cc:4841] TS 047b0407260f49dd99d40d89cd59fcad (127.3.233.130:36737): tablet 08386ea4082b47f1a0fa9b51836f4cbf (table test-table [id=dda68fc0edd148418a52d2d3061b992a]) successfully deleted
I20260619 16:50:16.255059 21547 ts_tablet_manager.cc:1918] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:16.255165 21547 log.cc:1196] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/08386ea4082b47f1a0fa9b51836f4cbf
I20260619 16:50:16.255476 21547 ts_tablet_manager.cc:1939] T 08386ea4082b47f1a0fa9b51836f4cbf P c9a11e4e892d4269bb945aed2a3763ce: Deleting consensus metadata
I20260619 16:50:16.256453 21030 catalog_manager.cc:4841] TS c9a11e4e892d4269bb945aed2a3763ce (127.3.233.131:39541): tablet 08386ea4082b47f1a0fa9b51836f4cbf (table test-table [id=dda68fc0edd148418a52d2d3061b992a]) successfully deleted
W20260619 16:50:16.256585 21032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:34159 (error 108)
I20260619 16:50:16.257426 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21242
I20260619 16:50:16.257542 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.257256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:34159 (local address 127.0.0.1:49338)
0619 16:50:16.257393 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.257459 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:34159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:50:16.258621 21032 catalog_manager.cc:4584] TS a3e09cb1a5804dd58a2aa15f3b794358 (127.3.233.129:34159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08386ea4082b47f1a0fa9b51836f4cbf: Network error: Client connection negotiation failed: client connection to 127.3.233.129:34159: connect: Connection refused (error 111)
I20260619 16:50:16.263262 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21376
I20260619 16:50:16.268887 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21014
2026-06-19T16:50:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2776 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T16:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:16Z Disabled control of system clock
I20260619 16:50:16.300526 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35253
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:50:16.412252 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:16.412550 21552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:16.412587 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:16.416658 21552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:16.416725 21552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:16.416740 21552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:16.416756 21552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:16.416772 21552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:16.421722 21552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:35253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:16.422801 21552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:16.424280 21552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:50:16.433245 21552 server_base.cc:1034] running on GCE node
W20260619 16:50:16.433270 21558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:16.433272 21560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:16.433244 21557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:16.434089 21552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:16.435331 21552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:16.436507 21552 hybrid_clock.cc:648] HybridClock initialized: now 1781887816436469 us; error 52 us; skew 500 ppm
I20260619 16:50:16.438442 21552 webserver.cc:460] Webserver started at http://127.3.233.190:45233/ using document root <none> and password file <none>
I20260619 16:50:16.439116 21552 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:16.439191 21552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:16.439419 21552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:16.441493 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "2e148c9c334842399bdcc3d6b2aafad3"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "7cd9700f866cb85939510362724bc31d"
server_key_iv: "b8abb3d1431b990c4d4f98f176eab8d0"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.442061 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "2e148c9c334842399bdcc3d6b2aafad3"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "7cd9700f866cb85939510362724bc31d"
server_key_iv: "b8abb3d1431b990c4d4f98f176eab8d0"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.445736 21552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 16:50:16.448246 21566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:16.449471 21552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:16.449608 21552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "2e148c9c334842399bdcc3d6b2aafad3"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "7cd9700f866cb85939510362724bc31d"
server_key_iv: "b8abb3d1431b990c4d4f98f176eab8d0"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.449719 21552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:16.468305 21552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:16.469081 21552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:16.469290 21552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:16.476774 21552 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:35253
I20260619 16:50:16.476780 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:35253 every 8 connection(s)
I20260619 16:50:16.477767 21552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:16.481026 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:16.487069 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21552
I20260619 16:50:16.487231 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:16.486737 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3: Bootstrap starting.
I20260619 16:50:16.487555 4006 external_mini_cluster.cc:1428] Setting key 56f35a25ac469273137b29485861e937
I20260619 16:50:16.490301 21619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:16.491179 21619 log.cc:824] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:16.493084 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3: No bootstrap required, opened a new log
I20260619 16:50:16.497677 21619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e148c9c334842399bdcc3d6b2aafad3" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35253 } }
I20260619 16:50:16.497905 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.489155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.3.233.190:35253)
0619 16:50:16.489952 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.489975 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.490033 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:50:16.490776 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.490796 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.491308 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.491680 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:16.492107 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.493121 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.494416 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.494893 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.495099 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.496017 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:16.496023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:16.496036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:16.496484 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:16.496543 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:16.496563 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:16.496637 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:16.496825 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 16:50:16.497111 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":135,"threads_started":1}
I20260619 16:50:16.497941 21619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:16.498016 21619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e148c9c334842399bdcc3d6b2aafad3, State: Initialized, Role: FOLLOWER
I20260619 16:50:16.498512 21619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e148c9c334842399bdcc3d6b2aafad3" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35253 } }
I20260619 16:50:16.498667 21619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:16.498754 21619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:16.498860 21619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:16.500085 21619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e148c9c334842399bdcc3d6b2aafad3" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35253 } }
I20260619 16:50:16.500460 21619 leader_election.cc:302] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e148c9c334842399bdcc3d6b2aafad3; no voters:
I20260619 16:50:16.500794 21619 leader_election.cc:288] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:16.500957 21624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:16.501264 21624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [term 1 LEADER]: Becoming Leader. State: Replica: 2e148c9c334842399bdcc3d6b2aafad3, State: Running, Role: LEADER
I20260619 16:50:16.501576 21624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e148c9c334842399bdcc3d6b2aafad3" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35253 } }
I20260619 16:50:16.502075 21619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:16.503238 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e148c9c334842399bdcc3d6b2aafad3. Latest consensus state: current_term: 1 leader_uuid: "2e148c9c334842399bdcc3d6b2aafad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e148c9c334842399bdcc3d6b2aafad3" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35253 } } }
I20260619 16:50:16.503369 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:16.503777 21633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:16.503779 21625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e148c9c334842399bdcc3d6b2aafad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e148c9c334842399bdcc3d6b2aafad3" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35253 } } }
I20260619 16:50:16.504040 21625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:16.506966 21633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:16.512426 21633 catalog_manager.cc:1347] Generated new cluster ID: a4a1c97eed17420587524a199a20992f
I20260619 16:50:16.512533 21633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:16.544987 21633 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:16.546159 21633 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:16.553262 21633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3: Generated new TSK 0
I20260619 16:50:16.554147 21633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:16.563077 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35253
--builtin_ntp_servers=127.3.233.148:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:50:16.681407 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:16.681715 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:16.681757 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:16.686029 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:16.686116 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:16.686177 21643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:50:16.691231 21643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:16.692375 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:16.693779 21643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:16.703711 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:16.703711 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:16.703778 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:16.704133 21643 server_base.cc:1034] running on GCE node
I20260619 16:50:16.704577 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:16.705309 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:16.706487 21643 hybrid_clock.cc:648] HybridClock initialized: now 1781887816706467 us; error 30 us; skew 500 ppm
I20260619 16:50:16.708580 21643 webserver.cc:460] Webserver started at http://127.3.233.129:46499/ using document root <none> and password file <none>
I20260619 16:50:16.709277 21643 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:16.709380 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:16.709618 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:16.711786 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "e6879d1c7f1346bfa8014d8002117262"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "970c67b476cd7d001879f5b8b10ba0f5"
server_key_iv: "83770f3de18dcb17f66652e14012cba4"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.712338 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "e6879d1c7f1346bfa8014d8002117262"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "970c67b476cd7d001879f5b8b10ba0f5"
server_key_iv: "83770f3de18dcb17f66652e14012cba4"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.716143 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:50:16.718684 21657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:16.720124 21643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:16.720327 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "e6879d1c7f1346bfa8014d8002117262"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "970c67b476cd7d001879f5b8b10ba0f5"
server_key_iv: "83770f3de18dcb17f66652e14012cba4"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.720451 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:16.732378 21643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:16.733078 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:16.733316 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:16.733901 21643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:16.734987 21643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:16.735065 21643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:16.735132 21643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:16.735186 21643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:16.744534 21643 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:45105
I20260619 16:50:16.744555 21770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:45105 every 8 connection(s)
I20260619 16:50:16.745525 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:16.749671 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21643
I20260619 16:50:16.749852 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:16.750173 4006 external_mini_cluster.cc:1428] Setting key bd264d9e5ce7572a3253df929b218adf
I20260619 16:50:16.752441 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35253
--builtin_ntp_servers=127.3.233.148:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:50:16.755836 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.747531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:55759 (local address 127.3.233.190:35253)
0619 16:50:16.747773 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.747777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.748757 (+ 980us) server_negotiation.cc:407] Connection header received
0619 16:50:16.750364 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.750368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.750439 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.750532 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:16.751702 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.752240 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.753596 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.753915 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.754026 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.755335 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:16.755341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:16.755348 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:16.755368 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:16.755389 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:16.755397 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:16.755472 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:16.755639 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:50:16.755699 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260619 16:50:16.756883 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.747939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35253 (local address 127.3.233.129:55759)
0619 16:50:16.748589 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.748633 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:16.750101 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:16.750724 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:16.750732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:16.750823 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:16.751554 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.751571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.752370 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.752374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.753471 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.753480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.754072 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.754076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.754260 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.754997 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:16.755001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:16.755057 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:16.755174 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:16.755486 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:16.755499 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:16.755509 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:50:16.755638 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:50:16.756020 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260619 16:50:16.758349 21771 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35253
I20260619 16:50:16.758615 21771 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:16.759240 21771 heartbeater.cc:511] Master 127.3.233.190:35253 requested a full tablet report, sending...
I20260619 16:50:16.760823 21583 ts_manager.cc:194] Registered new tserver with Master: e6879d1c7f1346bfa8014d8002117262 (127.3.233.129:45105)
I20260619 16:50:16.761935 21583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.129:55759
W20260619 16:50:16.875670 21774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:16.876013 21774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:16.876096 21774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:16.880759 21774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:16.880848 21774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:16.880947 21774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:50:16.886266 21774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:16.887528 21774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:16.888988 21774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:16.899102 21780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:16.899152 21782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:16.899156 21779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:16.899477 21774 server_base.cc:1034] running on GCE node
I20260619 16:50:16.900486 21774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:16.901091 21774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:16.902282 21774 hybrid_clock.cc:648] HybridClock initialized: now 1781887816902239 us; error 58 us; skew 500 ppm
I20260619 16:50:16.904400 21774 webserver.cc:460] Webserver started at http://127.3.233.130:43347/ using document root <none> and password file <none>
I20260619 16:50:16.904996 21774 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:16.905093 21774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:16.905387 21774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:16.907557 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "988ece0c6d2b412db9163717c81ef6a8"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "c59bce8f83a83d3d3631c618b37b1f29"
server_key_iv: "0e877403416d74185835ec165de2afe9"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.908167 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "988ece0c6d2b412db9163717c81ef6a8"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "c59bce8f83a83d3d3631c618b37b1f29"
server_key_iv: "0e877403416d74185835ec165de2afe9"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.912081 21774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:50:16.914700 21788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:16.916282 21774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:50:16.916460 21774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "988ece0c6d2b412db9163717c81ef6a8"
format_stamp: "Formatted at 2026-06-19 16:50:16 on dist-test-slave-cgwx"
server_key: "c59bce8f83a83d3d3631c618b37b1f29"
server_key_iv: "0e877403416d74185835ec165de2afe9"
server_key_version: "encryptionkey@0"
I20260619 16:50:16.916599 21774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:16.936125 21774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:16.936852 21774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:16.937058 21774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:16.937654 21774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:16.938601 21774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:16.938679 21774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:16.938755 21774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:16.938799 21774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:16.948297 21774 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:36187
I20260619 16:50:16.948328 21901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:36187 every 8 connection(s)
I20260619 16:50:16.949222 21774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:16.949685 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21774
I20260619 16:50:16.949810 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:16.950122 4006 external_mini_cluster.cc:1428] Setting key efb1e4a5a98217171c1bec3299513503
I20260619 16:50:16.953982 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35253
--builtin_ntp_servers=127.3.233.148:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:50:16.961274 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.951139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:42203 (local address 127.3.233.190:35253)
0619 16:50:16.951285 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:16.951289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:16.952869 (+ 1580us) server_negotiation.cc:407] Connection header received
0619 16:50:16.955061 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:16.955065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:16.955150 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:16.955272 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:16.956577 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.957443 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.959208 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.959443 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.959560 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.960766 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:16.960770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:16.960772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:16.960787 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:16.960815 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:16.960823 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:16.960901 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:16.961061 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:50:16.961122 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:50:16.962340 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:16.951731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35253 (local address 127.3.233.130:42203)
0619 16:50:16.952703 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:16.952761 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:16.954762 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:16.955520 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:16.955528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:16.955683 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:16.956384 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.956407 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.957599 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.957602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.959044 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:16.959056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:16.959603 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:16.959607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:16.959784 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:16.960482 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:16.960483 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:16.960526 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:16.960592 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:16.960902 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:16.960913 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:16.960920 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:50:16.961062 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:50:16.961580 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260619 16:50:16.963809 21902 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35253
I20260619 16:50:16.964054 21902 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:16.964577 21902 heartbeater.cc:511] Master 127.3.233.190:35253 requested a full tablet report, sending...
I20260619 16:50:16.965695 21583 ts_manager.cc:194] Registered new tserver with Master: 988ece0c6d2b412db9163717c81ef6a8 (127.3.233.130:36187)
I20260619 16:50:16.966334 21583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.130:42203
W20260619 16:50:17.070482 21905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:17.070892 21905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:17.071009 21905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:17.075297 21905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:17.075399 21905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:17.075493 21905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:50:17.080578 21905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:17.081781 21905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:17.083302 21905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:50:17.093662 21905 server_base.cc:1034] running on GCE node
W20260619 16:50:17.093631 21910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:17.093623 21913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:17.093623 21911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:17.094331 21905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:17.095173 21905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:17.096369 21905 hybrid_clock.cc:648] HybridClock initialized: now 1781887817096343 us; error 45 us; skew 500 ppm
I20260619 16:50:17.098431 21905 webserver.cc:460] Webserver started at http://127.3.233.131:34645/ using document root <none> and password file <none>
I20260619 16:50:17.099126 21905 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:17.099203 21905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:17.099440 21905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:17.101565 21905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "88335c70d6af4f96962997908b59393f"
format_stamp: "Formatted at 2026-06-19 16:50:17 on dist-test-slave-cgwx"
server_key: "2215c990504b05f288bbc8e99bc1f780"
server_key_iv: "8ace85bbb5750ee3e35209444ede8782"
server_key_version: "encryptionkey@0"
I20260619 16:50:17.102167 21905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "88335c70d6af4f96962997908b59393f"
format_stamp: "Formatted at 2026-06-19 16:50:17 on dist-test-slave-cgwx"
server_key: "2215c990504b05f288bbc8e99bc1f780"
server_key_iv: "8ace85bbb5750ee3e35209444ede8782"
server_key_version: "encryptionkey@0"
I20260619 16:50:17.106338 21905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 16:50:17.109050 21919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:17.110445 21905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:17.110591 21905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "88335c70d6af4f96962997908b59393f"
format_stamp: "Formatted at 2026-06-19 16:50:17 on dist-test-slave-cgwx"
server_key: "2215c990504b05f288bbc8e99bc1f780"
server_key_iv: "8ace85bbb5750ee3e35209444ede8782"
server_key_version: "encryptionkey@0"
I20260619 16:50:17.110709 21905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:17.121026 21905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:17.121809 21905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:17.122020 21905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:17.122620 21905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:17.123649 21905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:17.123696 21905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:17.123735 21905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:17.123790 21905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:17.132752 21905 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:36173
I20260619 16:50:17.132776 22032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:36173 every 8 connection(s)
I20260619 16:50:17.133646 21905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:17.142055 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 21905
I20260619 16:50:17.142187 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:17.142464 4006 external_mini_cluster.cc:1428] Setting key 083fe3ba7a612fd8a291e2c3b1ebddaa
I20260619 16:50:17.143668 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.135473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:53019 (local address 127.3.233.190:35253)
0619 16:50:17.135621 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.135627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.136749 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 16:50:17.138091 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.138093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.138146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.138222 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:17.139373 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.139894 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.141335 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.141573 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.141672 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.143115 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:17.143119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:17.143121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:17.143140 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:17.143162 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:17.143171 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:17.143268 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.143418 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.143504 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:50:17.144593 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.135951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:35253 (local address 127.3.233.131:53019)
0619 16:50:17.136592 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.136633 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.137884 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.138432 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.138439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.138527 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:17.139208 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.139228 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.140031 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.140034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.141075 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.141086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.141714 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.141717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.141957 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.142727 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:17.142730 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:17.142781 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:17.142870 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:17.143266 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:17.143274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:17.143280 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:50:17.143393 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:50:17.143698 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260619 16:50:17.145638 22033 heartbeater.cc:347] Connected to a master server at 127.3.233.190:35253
I20260619 16:50:17.145896 22033 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:17.146432 22033 heartbeater.cc:511] Master 127.3.233.190:35253 requested a full tablet report, sending...
I20260619 16:50:17.147475 21583 ts_manager.cc:194] Registered new tserver with Master: 88335c70d6af4f96962997908b59393f (127.3.233.131:36173)
I20260619 16:50:17.148037 21583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.131:53019
I20260619 16:50:17.157054 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:17.162963 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.158982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.3.233.190:35253)
0619 16:50:17.159133 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.159137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.159317 (+ 180us) server_negotiation.cc:407] Connection header received
0619 16:50:17.159472 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.159475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.159525 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.159610 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:17.159891 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.160423 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.161279 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.161498 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.161614 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.162448 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:17.162452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:17.162455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:17.162468 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:17.162494 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:17.162501 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:17.162577 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.162697 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.162764 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:50:17.165828 21583 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:17.169284 21583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:17.183924 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.177700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:36173 (local address 127.0.0.1:48700)
0619 16:50:17.178816 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.178865 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.179025 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.179652 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.179658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.179680 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:17.179825 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.179831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.180917 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.180921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.182225 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.182235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.183333 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.183337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.183473 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.183527 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:50:17.183632 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:50:17.183764 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260619 16:50:17.184271 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.177886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.3.233.131:36173)
0619 16:50:17.178960 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.178977 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.179030 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:50:17.179238 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.179244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.179499 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.179655 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:17.179940 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.180795 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.182352 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.183210 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.183315 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.183506 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.184004 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.184104 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":160,"threads_started":1}
I20260619 16:50:17.184348 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.178423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:36187 (local address 127.0.0.1:40122)
0619 16:50:17.179254 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.179267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.179362 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.179972 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.179975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.179997 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:17.180094 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.180100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.181531 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.181536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.182719 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.182729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.183960 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.183964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.184053 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.184073 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:50:17.184140 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:50:17.184215 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":102,"thread_start_us":117,"threads_started":1}
I20260619 16:50:17.184690 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.178437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.3.233.129:45105)
0619 16:50:17.179531 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.179541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.179561 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:50:17.179624 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.179628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.179816 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.180234 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:17.180365 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.181480 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.182811 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.183476 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.183582 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.183771 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.183821 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.183909 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":110,"threads_started":1}
I20260619 16:50:17.184777 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.178849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.3.233.130:36187)
0619 16:50:17.179230 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.179244 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.179330 (+ 86us) server_negotiation.cc:407] Connection header received
0619 16:50:17.179477 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.179483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.179678 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.179838 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:17.180225 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.181388 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.182866 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.183635 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.183731 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.183901 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.184542 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.184633 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":140,"threads_started":1}
I20260619 16:50:17.185189 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.177881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:45105 (local address 127.0.0.1:33898)
0619 16:50:17.179162 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.179204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.179296 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.179963 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.179968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.179984 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:17.180082 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.180086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.181589 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.181593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.182691 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.182705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.183586 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.183589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.183702 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.183728 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:50:17.183786 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 16:50:17.183851 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"mutex_wait_us":93,"thread_start_us":107,"threads_started":1}
I20260619 16:50:17.187477 21705 tablet_service.cc:1467] Processing CreateTablet for tablet 93584569443f44a0bc61c977618727a3 (DEFAULT_TABLE table=test-table [id=c924b2245153435ea3da113eb30cf1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:17.187489 21967 tablet_service.cc:1467] Processing CreateTablet for tablet 93584569443f44a0bc61c977618727a3 (DEFAULT_TABLE table=test-table [id=c924b2245153435ea3da113eb30cf1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:17.187490 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 93584569443f44a0bc61c977618727a3 (DEFAULT_TABLE table=test-table [id=c924b2245153435ea3da113eb30cf1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:17.188449 21705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93584569443f44a0bc61c977618727a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:17.188480 21967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93584569443f44a0bc61c977618727a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:17.188480 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93584569443f44a0bc61c977618727a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:17.194834 22050 tablet_bootstrap.cc:492] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262: Bootstrap starting.
I20260619 16:50:17.196323 22051 tablet_bootstrap.cc:492] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Bootstrap starting.
I20260619 16:50:17.197417 22050 tablet_bootstrap.cc:654] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:17.198150 22050 log.cc:824] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:17.198173 22051 tablet_bootstrap.cc:654] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:17.198813 22051 log.cc:824] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:17.199566 22052 tablet_bootstrap.cc:492] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Bootstrap starting.
I20260619 16:50:17.200225 22050 tablet_bootstrap.cc:492] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262: No bootstrap required, opened a new log
I20260619 16:50:17.200414 22050 ts_tablet_manager.cc:1397] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:50:17.201411 22052 tablet_bootstrap.cc:654] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:17.202121 22052 log.cc:824] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:17.204162 22050 raft_consensus.cc:348] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.204376 22050 raft_consensus.cc:374] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:17.204442 22050 raft_consensus.cc:729] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6879d1c7f1346bfa8014d8002117262, State: Initialized, Role: FOLLOWER
I20260619 16:50:17.204583 22052 tablet_bootstrap.cc:492] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: No bootstrap required, opened a new log
I20260619 16:50:17.204787 22052 ts_tablet_manager.cc:1397] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:50:17.204914 22050 consensus_queue.cc:260] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.205515 21771 heartbeater.cc:503] Master 127.3.233.190:35253 was elected leader, sending a full tablet report...
I20260619 16:50:17.205899 22050 ts_tablet_manager.cc:1428] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 16:50:17.207557 22051 tablet_bootstrap.cc:492] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: No bootstrap required, opened a new log
I20260619 16:50:17.207795 22051 ts_tablet_manager.cc:1397] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260619 16:50:17.208639 22052 raft_consensus.cc:348] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.208909 22052 raft_consensus.cc:374] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:17.208971 22052 raft_consensus.cc:729] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 988ece0c6d2b412db9163717c81ef6a8, State: Initialized, Role: FOLLOWER
I20260619 16:50:17.209403 22052 consensus_queue.cc:260] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.210031 21902 heartbeater.cc:503] Master 127.3.233.190:35253 was elected leader, sending a full tablet report...
I20260619 16:50:17.210302 22052 ts_tablet_manager.cc:1428] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 16:50:17.211580 22051 raft_consensus.cc:348] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.211820 22051 raft_consensus.cc:374] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:17.211879 22051 raft_consensus.cc:729] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88335c70d6af4f96962997908b59393f, State: Initialized, Role: FOLLOWER
I20260619 16:50:17.212265 22051 consensus_queue.cc:260] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.212908 22033 heartbeater.cc:503] Master 127.3.233.190:35253 was elected leader, sending a full tablet report...
I20260619 16:50:17.213235 22051 ts_tablet_manager.cc:1428] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 16:50:17.246851 21772 tablet.cc:2369] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:17.295109 22058 raft_consensus.cc:482] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:17.295344 22058 raft_consensus.cc:504] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.296666 22058 leader_election.cc:288] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6879d1c7f1346bfa8014d8002117262 (127.3.233.129:45105), 988ece0c6d2b412db9163717c81ef6a8 (127.3.233.130:36187)
I20260619 16:50:17.301548 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.297298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:49549 (local address 127.3.233.129:45105)
0619 16:50:17.297418 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.297422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.297585 (+ 163us) server_negotiation.cc:407] Connection header received
0619 16:50:17.297688 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.297691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.297760 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.297853 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:17.298248 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.298903 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.300485 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.300892 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.301033 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.301086 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.301337 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.301410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:50:17.301563 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.297192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:51239 (local address 127.3.233.130:36187)
0619 16:50:17.297329 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.297334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.297349 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:50:17.297407 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.297409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.297458 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.297558 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:17.297926 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.298787 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.300455 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.300914 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.301056 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.301106 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.301359 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.301435 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:50:17.301563 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.296954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:36187 (local address 127.3.233.131:51239)
0619 16:50:17.297121 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.297137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.297239 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.297609 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.297614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.297640 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:17.297779 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.297801 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.298965 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.298968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.300308 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.300319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.301045 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.301049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.301170 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.301249 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 16:50:17.301339 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:50:17.301425 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":47}
I20260619 16:50:17.301594 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.296955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:45105 (local address 127.3.233.131:49549)
0619 16:50:17.297491 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.297507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:17.297595 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:17.297909 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:17.297912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:17.297932 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:17.298058 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.298064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.299017 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.299020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.300308 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:17.300319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.301028 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.301032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:17.301133 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.301227 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 16:50:17.301318 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:50:17.301395 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":184,"thread_start_us":125,"threads_started":1}
I20260619 16:50:17.302297 21725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93584569443f44a0bc61c977618727a3" candidate_uuid: "88335c70d6af4f96962997908b59393f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6879d1c7f1346bfa8014d8002117262" is_pre_election: true
I20260619 16:50:17.302350 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93584569443f44a0bc61c977618727a3" candidate_uuid: "88335c70d6af4f96962997908b59393f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "988ece0c6d2b412db9163717c81ef6a8" is_pre_election: true
I20260619 16:50:17.302655 21725 raft_consensus.cc:2454] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88335c70d6af4f96962997908b59393f in term 0.
I20260619 16:50:17.302670 21856 raft_consensus.cc:2454] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88335c70d6af4f96962997908b59393f in term 0.
I20260619 16:50:17.303323 21923 leader_election.cc:302] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 88335c70d6af4f96962997908b59393f, 988ece0c6d2b412db9163717c81ef6a8, e6879d1c7f1346bfa8014d8002117262; no voters:
I20260619 16:50:17.303644 22058 raft_consensus.cc:2781] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:17.303735 22058 raft_consensus.cc:482] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:17.303781 22058 raft_consensus.cc:3037] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:17.304909 22058 raft_consensus.cc:504] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.305255 22058 leader_election.cc:288] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [CANDIDATE]: Term 1 election: Requested vote from peers e6879d1c7f1346bfa8014d8002117262 (127.3.233.129:45105), 988ece0c6d2b412db9163717c81ef6a8 (127.3.233.130:36187)
I20260619 16:50:17.305624 21725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93584569443f44a0bc61c977618727a3" candidate_uuid: "88335c70d6af4f96962997908b59393f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6879d1c7f1346bfa8014d8002117262"
I20260619 16:50:17.305660 21856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93584569443f44a0bc61c977618727a3" candidate_uuid: "88335c70d6af4f96962997908b59393f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "988ece0c6d2b412db9163717c81ef6a8"
I20260619 16:50:17.305732 21725 raft_consensus.cc:3037] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:17.305848 21856 raft_consensus.cc:3037] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:17.306813 21725 raft_consensus.cc:2454] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88335c70d6af4f96962997908b59393f in term 1.
I20260619 16:50:17.306967 21856 raft_consensus.cc:2454] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88335c70d6af4f96962997908b59393f in term 1.
I20260619 16:50:17.307158 21921 leader_election.cc:302] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88335c70d6af4f96962997908b59393f, e6879d1c7f1346bfa8014d8002117262; no voters:
I20260619 16:50:17.307384 22058 raft_consensus.cc:2781] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:17.307605 22058 raft_consensus.cc:686] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 1 LEADER]: Becoming Leader. State: Replica: 88335c70d6af4f96962997908b59393f, State: Running, Role: LEADER
I20260619 16:50:17.307988 22058 consensus_queue.cc:237] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } }
I20260619 16:50:17.311240 21577 catalog_manager.cc:5495] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f reported cstate change: term changed from 0 to 1, leader changed from <none> to 88335c70d6af4f96962997908b59393f (127.3.233.131). New cstate: current_term: 1 leader_uuid: "88335c70d6af4f96962997908b59393f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88335c70d6af4f96962997908b59393f" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 36173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 } health_report { overall_health: UNKNOWN } } }
I20260619 16:50:17.336877 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.333470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.3.233.131:36173)
0619 16:50:17.333594 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:17.333598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:17.333614 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:17.333712 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:17.333714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:17.333770 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:17.333860 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:17.334067 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.334556 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.335439 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:17.335651 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:17.335744 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:17.336023 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:17.336452 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:17.336542 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:17.336697 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 16:50:17.336745 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:50:17.342916 21856 raft_consensus.cc:1261] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 1 FOLLOWER]: Refusing update from remote peer 88335c70d6af4f96962997908b59393f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:17.342900 21725 raft_consensus.cc:1261] T 93584569443f44a0bc61c977618727a3 P e6879d1c7f1346bfa8014d8002117262 [term 1 FOLLOWER]: Refusing update from remote peer 88335c70d6af4f96962997908b59393f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:17.343714 22060 consensus_queue.cc:1035] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6879d1c7f1346bfa8014d8002117262" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:17.343905 22058 consensus_queue.cc:1035] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [LEADER]: Connected to new peer: Peer: permanent_uuid: "988ece0c6d2b412db9163717c81ef6a8" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:17.352233 22064 mvcc.cc:205] Tried to move back new op lower bound from 7298612499828387840 to 7298612499696201728. Current Snapshot: MvccSnapshot[applied={T|T < 7298612499828387840}]
I20260619 16:50:17.352424 22063 mvcc.cc:205] Tried to move back new op lower bound from 7298612499828387840 to 7298612499696201728. Current Snapshot: MvccSnapshot[applied={T|T < 7298612499828387840}]
I20260619 16:50:17.352962 22066 mvcc.cc:205] Tried to move back new op lower bound from 7298612499828387840 to 7298612499696201728. Current Snapshot: MvccSnapshot[applied={T|T < 7298612499828387840}]
I20260619 16:50:17.360813 21577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:17.360982 21577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:17.363828 21577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e148c9c334842399bdcc3d6b2aafad3: Sending DeleteTablet for 3 replicas of tablet 93584569443f44a0bc61c977618727a3
I20260619 16:50:17.364707 21967 tablet_service.cc:1514] Processing DeleteTablet for tablet 93584569443f44a0bc61c977618727a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:17 UTC) from {username='slave'} at 127.0.0.1:48700
I20260619 16:50:17.364667 21705 tablet_service.cc:1514] Processing DeleteTablet for tablet 93584569443f44a0bc61c977618727a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:17 UTC) from {username='slave'} at 127.0.0.1:33898
I20260619 16:50:17.364750 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 93584569443f44a0bc61c977618727a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:17 UTC) from {username='slave'} at 127.0.0.1:40122
I20260619 16:50:17.365401 22073 tablet_replica.cc:330] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: stopping tablet replica
I20260619 16:50:17.365553 22074 tablet_replica.cc:330] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: stopping tablet replica
I20260619 16:50:17.365737 22073 raft_consensus.cc:2229] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:17.365872 22074 raft_consensus.cc:2229] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:17.366187 22073 raft_consensus.cc:2258] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:17.366272 22074 raft_consensus.cc:2258] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:17.366336 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21643
I20260619 16:50:17.368506 22073 ts_tablet_manager.cc:1905] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:17.372404 22074 ts_tablet_manager.cc:1905] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:17.372470 22073 ts_tablet_manager.cc:1918] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:17.372593 22073 log.cc:1196] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/93584569443f44a0bc61c977618727a3
I20260619 16:50:17.372956 22073 ts_tablet_manager.cc:1939] T 93584569443f44a0bc61c977618727a3 P 88335c70d6af4f96962997908b59393f: Deleting consensus metadata
I20260619 16:50:17.374562 21567 catalog_manager.cc:4841] TS 88335c70d6af4f96962997908b59393f (127.3.233.131:36173): tablet 93584569443f44a0bc61c977618727a3 (table test-table [id=c924b2245153435ea3da113eb30cf1f7]) successfully deleted
I20260619 16:50:17.375555 22074 ts_tablet_manager.cc:1918] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 16:50:17.375555 21568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:45105 (error 108)
I20260619 16:50:17.375681 22074 log.cc:1196] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/93584569443f44a0bc61c977618727a3
I20260619 16:50:17.376027 22074 ts_tablet_manager.cc:1939] T 93584569443f44a0bc61c977618727a3 P 988ece0c6d2b412db9163717c81ef6a8: Deleting consensus metadata
I20260619 16:50:17.376746 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.376432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:45105 (local address 127.0.0.1:33914)
0619 16:50:17.376569 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.376645 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:50:17.376796 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21774
W20260619 16:50:17.377024 21568 catalog_manager.cc:4584] TS e6879d1c7f1346bfa8014d8002117262 (127.3.233.129:45105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93584569443f44a0bc61c977618727a3: Network error: Client connection negotiation failed: client connection to 127.3.233.129:45105: connect: Connection refused (error 111)
I20260619 16:50:17.384294 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21905
I20260619 16:50:17.385922 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:17.385634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:36187 (local address 127.0.0.1:40136)
0619 16:50:17.385794 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:17.385845 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.130:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:50:17.390153 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 21552
2026-06-19T16:50:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1119 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:17 dist-test-slave-cgwx krb5kdc[22078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:17 dist-test-slave-cgwx krb5kdc[22078](info): set up 2 sockets
Jun 19 16:50:17 dist-test-slave-cgwx krb5kdc[22078](info): commencing operation
krb5kdc: starting...
W20260619 16:50:18.430868 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:18 dist-test-slave-cgwx krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:18.562036 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:45379
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:18.685678 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:18.685949 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:18.685990 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:18.690153 22091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:18.690217 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:18.690236 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:18.690253 22091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:18.690268 22091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:18.695721 22091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:45379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:18.696830 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:18.697888 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:18.707005 22097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:18.706996 22099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:18.707048 22091 server_base.cc:1034] running on GCE node
W20260619 16:50:18.706998 22096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:18.707885 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:18.709038 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:18.710268 22091 hybrid_clock.cc:648] HybridClock initialized: now 1781887818710191 us; error 100 us; skew 500 ppm
Jun 19 16:50:18 dist-test-slave-cgwx krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887818, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:18.716887 22091 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:18.717881 22091 webserver.cc:460] Webserver started at http://127.3.233.190:34381/ using document root <none> and password file <none>
I20260619 16:50:18.718494 22091 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:18.718585 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:18.718814 22091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:18.720932 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "331e29e2a09d4b018cec44006a68e01d"
format_stamp: "Formatted at 2026-06-19 16:50:18 on dist-test-slave-cgwx"
server_key: "0c72a96daf1523dd7c028a4a9ef7cdc6"
server_key_iv: "15f2000c5c467842da5593ca12d9f2b7"
server_key_version: "encryptionkey@0"
I20260619 16:50:18.721496 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "331e29e2a09d4b018cec44006a68e01d"
format_stamp: "Formatted at 2026-06-19 16:50:18 on dist-test-slave-cgwx"
server_key: "0c72a96daf1523dd7c028a4a9ef7cdc6"
server_key_iv: "15f2000c5c467842da5593ca12d9f2b7"
server_key_version: "encryptionkey@0"
I20260619 16:50:18.725369 22091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:18.727977 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:18.729400 22091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:18.729547 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "331e29e2a09d4b018cec44006a68e01d"
format_stamp: "Formatted at 2026-06-19 16:50:18 on dist-test-slave-cgwx"
server_key: "0c72a96daf1523dd7c028a4a9ef7cdc6"
server_key_iv: "15f2000c5c467842da5593ca12d9f2b7"
server_key_version: "encryptionkey@0"
I20260619 16:50:18.729665 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:18.751343 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:18.754738 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:18.754971 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:18.762429 22091 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:45379
I20260619 16:50:18.762455 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:45379 every 8 connection(s)
I20260619 16:50:18.763474 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:18.766363 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:18.769524 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22091
I20260619 16:50:18.769672 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:18.769992 4006 external_mini_cluster.cc:1428] Setting key 26588347853f09f75628a060b4dde7ec
I20260619 16:50:18.772634 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d: Bootstrap starting.
I20260619 16:50:18.775269 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:18.776090 22159 log.cc:824] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:18.778115 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d: No bootstrap required, opened a new log
Jun 19 16:50:18 dist-test-slave-cgwx krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:18.781522 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331e29e2a09d4b018cec44006a68e01d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45379 } }
I20260619 16:50:18.781780 22159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:18.781845 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331e29e2a09d4b018cec44006a68e01d, State: Initialized, Role: FOLLOWER
I20260619 16:50:18.782341 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331e29e2a09d4b018cec44006a68e01d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45379 } }
I20260619 16:50:18.782496 22159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:18.782577 22159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:18.782694 22159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:18.783633 22159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331e29e2a09d4b018cec44006a68e01d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45379 } }
I20260619 16:50:18.783963 22159 leader_election.cc:302] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 331e29e2a09d4b018cec44006a68e01d; no voters:
I20260619 16:50:18.784315 22159 leader_election.cc:288] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:18.784389 22164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:18.784596 22164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [term 1 LEADER]: Becoming Leader. State: Replica: 331e29e2a09d4b018cec44006a68e01d, State: Running, Role: LEADER
I20260619 16:50:18.784886 22164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331e29e2a09d4b018cec44006a68e01d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45379 } }
I20260619 16:50:18.785660 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:18.786034 22166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "331e29e2a09d4b018cec44006a68e01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331e29e2a09d4b018cec44006a68e01d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45379 } } }
I20260619 16:50:18.786047 22164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331e29e2a09d4b018cec44006a68e01d. Latest consensus state: current_term: 1 leader_uuid: "331e29e2a09d4b018cec44006a68e01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331e29e2a09d4b018cec44006a68e01d" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45379 } } }
I20260619 16:50:18.786321 22166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:18.786366 22164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:18.786484 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:18.771500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.3.233.190:45379)
0619 16:50:18.771944 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:18.771959 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:18.771995 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:50:18.772629 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:18.772645 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:18.773052 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:18.773343 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:18.774316 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.775829 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.776927 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:18.777418 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:18.777595 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:18.780493 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:18.780498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:18.780512 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:18.780538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:18.783031 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:18.783480 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:18.783486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:18.783491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:18.783612 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:18.783906 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:18.783909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:18.783911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:18.784443 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:18.784643 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:18.785036 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 16:50:18.785397 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260619 16:50:18.787058 22173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:18.790444 22173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:18.795573 22173 catalog_manager.cc:1347] Generated new cluster ID: c09bd7f02f714d19b89b385db7ec4b71
I20260619 16:50:18.795640 22173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:18.813287 22173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:18.814818 22173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:18.834045 22173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d: Generated new TSK 0
I20260619 16:50:18.834985 22173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:50:18.869699 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:45379
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:18.994568 22185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:18.994884 22185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:18.994993 22185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:18.999115 22185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:18.999205 22185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:18.999295 22185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:50:19.004729 22185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:19.005841 22185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:19.006896 22185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:50:19.017112 22185 server_base.cc:1034] running on GCE node
W20260619 16:50:19.017052 22193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:19.017019 22191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:19.017040 22190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:19.017674 22185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:19.018365 22185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:19.019549 22185 hybrid_clock.cc:648] HybridClock initialized: now 1781887819019529 us; error 33 us; skew 500 ppm
Jun 19 16:50:19 dist-test-slave-cgwx krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:19.022653 22185 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:50:19.023797 22185 webserver.cc:460] Webserver started at http://127.3.233.129:45987/ using document root <none> and password file <none>
I20260619 16:50:19.024430 22185 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:19.024497 22185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:19.024771 22185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:19.027201 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "fc0de85fe3eb4853bb6b7e429dff6951"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "a2230cb9af57b59b2113bcd5be7b7c37"
server_key_iv: "6b066d09e2fb9a806623d0cfc80d315a"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.027830 22185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "fc0de85fe3eb4853bb6b7e429dff6951"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "a2230cb9af57b59b2113bcd5be7b7c37"
server_key_iv: "6b066d09e2fb9a806623d0cfc80d315a"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.032832 22185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:50:19.036371 22200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.037750 22185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:50:19.037894 22185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "fc0de85fe3eb4853bb6b7e429dff6951"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "a2230cb9af57b59b2113bcd5be7b7c37"
server_key_iv: "6b066d09e2fb9a806623d0cfc80d315a"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.038023 22185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:19.068538 22185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:19.072184 22185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:19.072417 22185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:19.073071 22185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:19.073938 22185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:19.074011 22185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.074083 22185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:19.074126 22185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.083160 22185 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:37971
I20260619 16:50:19.083282 22313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:37971 every 8 connection(s)
I20260619 16:50:19.084108 22185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:19.088259 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22185
I20260619 16:50:19.088395 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:19.088738 4006 external_mini_cluster.cc:1428] Setting key 88092693857d9fb10b3996ff9451561d
Jun 19 16:50:19 dist-test-slave-cgwx krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:50:19.099311 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.086068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:57681 (local address 127.3.233.190:45379)
0619 16:50:19.086291 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.086298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.087279 (+ 981us) server_negotiation.cc:407] Connection header received
0619 16:50:19.089024 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.089027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.089078 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.089186 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:19.090414 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.090998 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.092167 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.092408 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.092519 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.095017 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:19.095023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:19.095029 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:19.095077 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:19.097014 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.097532 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.097536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.097539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.097638 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.097998 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.098004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.098006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.098323 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:19.098445 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.099013 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.099173 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:50:19.100255 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.086557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:45379 (local address 127.3.233.129:57681)
0619 16:50:19.087118 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.087163 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.088793 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.089380 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.089390 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.089746 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:19.090255 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.090268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.091120 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.091123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.092026 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.092035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.092561 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.092566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.092737 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.093355 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:19.093373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:19.094804 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:19.097138 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:19.097143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:19.097155 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:19.097418 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:19.097740 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:19.097742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:19.097743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:19.097896 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:19.098436 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:19.098441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:19.098679 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.098905 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.099241 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:50:19.101912 22314 heartbeater.cc:347] Connected to a master server at 127.3.233.190:45379
I20260619 16:50:19.102200 22314 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:19.103232 22314 heartbeater.cc:511] Master 127.3.233.190:45379 requested a full tablet report, sending...
I20260619 16:50:19.105082 22121 ts_manager.cc:194] Registered new tserver with Master: fc0de85fe3eb4853bb6b7e429dff6951 (127.3.233.129:37971)
I20260619 16:50:19.106318 22121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:57681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:50:19.120561 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:45379
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:19.233783 22319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:19.234095 22319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:19.234176 22319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:19.238305 22319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:19.238415 22319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:19.238510 22319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:50:19.244030 22319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:19.245194 22319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:19.246309 22319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:19.256162 22327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:19.256162 22324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:19.256289 22325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:19.256500 22319 server_base.cc:1034] running on GCE node
I20260619 16:50:19.256981 22319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:19.257814 22319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:19.259023 22319 hybrid_clock.cc:648] HybridClock initialized: now 1781887819259004 us; error 29 us; skew 500 ppm
Jun 19 16:50:19 dist-test-slave-cgwx krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:19.261924 22319 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:50:19.262995 22319 webserver.cc:460] Webserver started at http://127.3.233.130:39827/ using document root <none> and password file <none>
I20260619 16:50:19.263634 22319 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:19.263736 22319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:19.263967 22319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:19.266505 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "12dd338637c7437c9b274a571015f504"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "3793b0c2616373044b80cd4a4de9c42e"
server_key_iv: "829d10f6e6430f8472cc63fc7e21f3bb"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.267210 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "12dd338637c7437c9b274a571015f504"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "3793b0c2616373044b80cd4a4de9c42e"
server_key_iv: "829d10f6e6430f8472cc63fc7e21f3bb"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.271730 22319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:50:19.274641 22334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.276064 22319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:50:19.276232 22319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "12dd338637c7437c9b274a571015f504"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "3793b0c2616373044b80cd4a4de9c42e"
server_key_iv: "829d10f6e6430f8472cc63fc7e21f3bb"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.276433 22319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:19.289682 22319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:19.293308 22319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:19.293534 22319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:19.294180 22319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:19.295109 22319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:19.295179 22319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.295253 22319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:19.295306 22319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.304744 22319 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:39129
I20260619 16:50:19.304756 22447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:39129 every 8 connection(s)
I20260619 16:50:19.305683 22319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:19.308198 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22319
I20260619 16:50:19.308354 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:19.308710 4006 external_mini_cluster.cc:1428] Setting key 1db99ae84b49592e61aae76067c3ee04
Jun 19 16:50:19 dist-test-slave-cgwx krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
I20260619 16:50:19.324951 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.307701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:41839 (local address 127.3.233.190:45379)
0619 16:50:19.307856 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.307861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.308848 (+ 987us) server_negotiation.cc:407] Connection header received
0619 16:50:19.310589 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.310592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.310634 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.310700 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:19.312493 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.313373 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.315482 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.315708 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.315822 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.319716 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:19.319721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:19.319723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:19.319783 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:19.321791 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.322706 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.322713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.322719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.322838 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.323360 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.323409 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.323412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.323688 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:19.323814 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.324591 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.324739 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:50:19.325917 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.308131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:45379 (local address 127.3.233.130:41839)
0619 16:50:19.308667 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.308730 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.310357 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.310903 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.310916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.311537 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:19.312301 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.312340 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.313529 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.313536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.315282 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.315298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.315911 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.315919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.316238 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.317175 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:19.317205 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:19.319474 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:19.321964 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:19.321976 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:19.322001 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:19.322569 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:19.323003 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:19.323010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:19.323016 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:19.323241 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:19.323851 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:19.323859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:19.324227 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.324504 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.324906 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
I20260619 16:50:19.327708 22448 heartbeater.cc:347] Connected to a master server at 127.3.233.190:45379
I20260619 16:50:19.328007 22448 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:19.328645 22448 heartbeater.cc:511] Master 127.3.233.190:45379 requested a full tablet report, sending...
I20260619 16:50:19.329970 22121 ts_manager.cc:194] Registered new tserver with Master: 12dd338637c7437c9b274a571015f504 (127.3.233.130:39129)
I20260619 16:50:19.330823 22121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:41839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:50:19.342347 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:45379
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:19.457346 22453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:19.457630 22453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:19.457683 22453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:19.461719 22453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:19.461779 22453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:19.461839 22453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:50:19.467934 22453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:19.469094 22453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:19.470057 22453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:19.479982 22459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:19.480171 22453 server_base.cc:1034] running on GCE node
W20260619 16:50:19.480002 22461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:19.479964 22458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:19.480815 22453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:19.481581 22453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:19.483000 22453 hybrid_clock.cc:648] HybridClock initialized: now 1781887819482884 us; error 53 us; skew 500 ppm
Jun 19 16:50:19 dist-test-slave-cgwx krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:19.486352 22453 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:50:19.487399 22453 webserver.cc:460] Webserver started at http://127.3.233.131:39765/ using document root <none> and password file <none>
I20260619 16:50:19.488003 22453 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:19.488095 22453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:19.488341 22453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:19.490509 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "7f1ca91524e84960ad9b6dd3817dd2ed"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "3de473736dfb157bd6837a26a93bc4e4"
server_key_iv: "35ac4560fc9eb0bba441e9fe5cafc3d7"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.491144 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "7f1ca91524e84960ad9b6dd3817dd2ed"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "3de473736dfb157bd6837a26a93bc4e4"
server_key_iv: "35ac4560fc9eb0bba441e9fe5cafc3d7"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.495117 22453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:50:19.497777 22468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.499490 22453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:19.499616 22453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "7f1ca91524e84960ad9b6dd3817dd2ed"
format_stamp: "Formatted at 2026-06-19 16:50:19 on dist-test-slave-cgwx"
server_key: "3de473736dfb157bd6837a26a93bc4e4"
server_key_iv: "35ac4560fc9eb0bba441e9fe5cafc3d7"
server_key_version: "encryptionkey@0"
I20260619 16:50:19.499750 22453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:19.525043 22453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:19.528616 22453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:19.528839 22453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:19.529497 22453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:19.530360 22453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:19.530435 22453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.530511 22453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:19.530555 22453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:19.540644 22453 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:38767
I20260619 16:50:19.540665 22581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:38767 every 8 connection(s)
I20260619 16:50:19.541709 22453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:19.549947 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22453
I20260619 16:50:19.550128 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:19.550556 4006 external_mini_cluster.cc:1428] Setting key 17ce595947d13f51fca9500c8311eece
Jun 19 16:50:19 dist-test-slave-cgwx krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:19.557055 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.543781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:34477 (local address 127.3.233.190:45379)
0619 16:50:19.543951 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.543955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.544851 (+ 896us) server_negotiation.cc:407] Connection header received
0619 16:50:19.546288 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.546292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.546353 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.546434 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:19.547784 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.548350 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.549549 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.549771 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.549890 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.552984 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:19.552987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:19.552990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:19.553016 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:19.554561 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.555313 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.555318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.555320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.555418 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.555816 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.555820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.555822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.556038 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:19.556155 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.556726 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.556869 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:50:19.558009 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.544199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:45379 (local address 127.3.233.131:34477)
0619 16:50:19.544671 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.544717 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.546048 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.546617 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.546624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.547093 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:19.547625 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.547640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.548492 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.548496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.549408 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.549418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.549933 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.549937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.550151 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.551097 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:19.551113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:19.552785 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:19.554723 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:19.554730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:19.554781 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:19.555189 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:19.555523 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:19.555527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:19.555529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:19.555694 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:19.556154 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:19.556164 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:19.556459 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.556667 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.557051 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260619 16:50:19.559477 22582 heartbeater.cc:347] Connected to a master server at 127.3.233.190:45379
I20260619 16:50:19.559778 22582 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:19.560333 22582 heartbeater.cc:511] Master 127.3.233.190:45379 requested a full tablet report, sending...
I20260619 16:50:19.561554 22121 ts_manager.cc:194] Registered new tserver with Master: 7f1ca91524e84960ad9b6dd3817dd2ed (127.3.233.131:38767)
I20260619 16:50:19.562183 22121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:34477
I20260619 16:50:19.564875 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:19.575156 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.567047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.3.233.190:45379)
0619 16:50:19.567173 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.567177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.567385 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:50:19.567530 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.567533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.567582 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.567671 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:19.568429 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.568961 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.569854 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.570075 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.570186 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.571935 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:19.571939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:19.571941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:19.571978 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:19.573242 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.573789 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.573792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.573794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.573875 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:19.574202 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:19.574205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:19.574207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:19.574420 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:19.574580 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.574859 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.574985 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 16:50:19.578478 22121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:19.581815 22121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:19.597163 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.591164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39129 (local address 127.0.0.1:50576)
0619 16:50:19.591867 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.591880 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.591954 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.592968 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.592972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.592990 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:19.593121 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.593131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.594248 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.594252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.595333 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.595374 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.596583 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.596586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.596716 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.596756 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.596876 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.597009 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260619 16:50:19.597172 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.591823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.3.233.130:39129)
0619 16:50:19.592499 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.592513 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.592537 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:50:19.592614 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.592621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.592826 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.592992 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:19.593261 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.594102 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.595511 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.596458 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.596584 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.596755 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.596907 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.597009 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":140,"threads_started":1}
I20260619 16:50:19.597918 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.590970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:37971 (local address 127.0.0.1:41826)
0619 16:50:19.591702 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.591760 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.591906 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.592825 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.592831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.592855 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:19.593115 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.593141 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.595016 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.595019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.596311 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.596323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.597498 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.597503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.597594 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.597622 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.597675 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.597777 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":80,"thread_start_us":110,"threads_started":1}
I20260619 16:50:19.598189 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.591374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38767 (local address 127.0.0.1:52432)
0619 16:50:19.592514 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.592528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.592630 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.593224 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.593228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.593254 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:19.593334 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.593339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.595514 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.595520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.596499 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.596511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.597775 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.597779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.597865 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.597888 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.597977 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.598044 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260619 16:50:19.598604 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.591823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.3.233.131:38767)
0619 16:50:19.592374 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.592386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.592734 (+ 348us) server_negotiation.cc:407] Connection header received
0619 16:50:19.592821 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.592826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.593070 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.593263 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:19.593430 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.594783 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.596649 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.597653 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.597764 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.597900 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.598198 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.598466 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":140,"threads_started":1}
I20260619 16:50:19.599650 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.591170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.3.233.129:37971)
0619 16:50:19.592205 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.592220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.592247 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:50:19.592333 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.592339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.592586 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.593289 (+ 703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:19.593488 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.594783 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.596456 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.597343 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.598249 (+ 906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.598409 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.598459 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.599521 (+ 1062us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":139,"threads_started":1}
I20260619 16:50:19.600705 22382 tablet_service.cc:1467] Processing CreateTablet for tablet b4482fdab06549f493a011451f29305d (DEFAULT_TABLE table=test-table [id=e5532d15ff924ff8830a62b752fb206e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:19.601378 22516 tablet_service.cc:1467] Processing CreateTablet for tablet b4482fdab06549f493a011451f29305d (DEFAULT_TABLE table=test-table [id=e5532d15ff924ff8830a62b752fb206e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:19.601920 22382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4482fdab06549f493a011451f29305d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:19.601919 22248 tablet_service.cc:1467] Processing CreateTablet for tablet b4482fdab06549f493a011451f29305d (DEFAULT_TABLE table=test-table [id=e5532d15ff924ff8830a62b752fb206e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:19.602411 22516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4482fdab06549f493a011451f29305d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:19.602942 22248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4482fdab06549f493a011451f29305d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:19.609289 22599 tablet_bootstrap.cc:492] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: Bootstrap starting.
I20260619 16:50:19.610677 22600 tablet_bootstrap.cc:492] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Bootstrap starting.
I20260619 16:50:19.611279 22601 tablet_bootstrap.cc:492] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Bootstrap starting.
I20260619 16:50:19.611932 22599 tablet_bootstrap.cc:654] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:19.612870 22599 log.cc:824] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:19.613343 22601 tablet_bootstrap.cc:654] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:19.613415 22600 tablet_bootstrap.cc:654] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:19.614320 22600 log.cc:824] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:19.614578 22601 log.cc:824] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:19.615273 22599 tablet_bootstrap.cc:492] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: No bootstrap required, opened a new log
I20260619 16:50:19.615504 22599 ts_tablet_manager.cc:1397] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:50:19.616267 22600 tablet_bootstrap.cc:492] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: No bootstrap required, opened a new log
I20260619 16:50:19.616475 22600 ts_tablet_manager.cc:1397] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:50:19.617754 22601 tablet_bootstrap.cc:492] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: No bootstrap required, opened a new log
I20260619 16:50:19.617944 22601 ts_tablet_manager.cc:1397] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:50:19.619597 22599 raft_consensus.cc:348] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.619877 22599 raft_consensus.cc:374] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:19.619930 22599 raft_consensus.cc:729] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0de85fe3eb4853bb6b7e429dff6951, State: Initialized, Role: FOLLOWER
I20260619 16:50:19.620407 22599 consensus_queue.cc:260] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.620428 22600 raft_consensus.cc:348] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.620711 22600 raft_consensus.cc:374] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:19.620769 22600 raft_consensus.cc:729] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12dd338637c7437c9b274a571015f504, State: Initialized, Role: FOLLOWER
I20260619 16:50:19.621220 22600 consensus_queue.cc:260] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.621555 22599 ts_tablet_manager.cc:1428] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:50:19.621945 22448 heartbeater.cc:503] Master 127.3.233.190:45379 was elected leader, sending a full tablet report...
I20260619 16:50:19.621827 22601 raft_consensus.cc:348] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.622107 22601 raft_consensus.cc:374] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:19.622167 22601 raft_consensus.cc:729] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1ca91524e84960ad9b6dd3817dd2ed, State: Initialized, Role: FOLLOWER
I20260619 16:50:19.622283 22600 ts_tablet_manager.cc:1428] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:50:19.622531 22314 heartbeater.cc:503] Master 127.3.233.190:45379 was elected leader, sending a full tablet report...
I20260619 16:50:19.622639 22601 consensus_queue.cc:260] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.623638 22582 heartbeater.cc:503] Master 127.3.233.190:45379 was elected leader, sending a full tablet report...
I20260619 16:50:19.623741 22601 ts_tablet_manager.cc:1428] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260619 16:50:19.792757 22583 tablet.cc:2369] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:50:19.806859 22449 tablet.cc:2369] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:50:19.835636 22315 tablet.cc:2369] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:19.937326 22606 raft_consensus.cc:482] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:19.937629 22606 raft_consensus.cc:504] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.938897 22606 leader_election.cc:288] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc0de85fe3eb4853bb6b7e429dff6951 (127.3.233.129:37971), 7f1ca91524e84960ad9b6dd3817dd2ed (127.3.233.131:38767)
I20260619 16:50:19.939603 22605 raft_consensus.cc:482] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:19.939793 22605 raft_consensus.cc:504] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.941010 22605 leader_election.cc:288] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12dd338637c7437c9b274a571015f504 (127.3.233.130:39129), 7f1ca91524e84960ad9b6dd3817dd2ed (127.3.233.131:38767)
I20260619 16:50:19.943897 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.939314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:52407 (local address 127.3.233.131:38767)
0619 16:50:19.939464 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.939469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.939707 (+ 238us) server_negotiation.cc:407] Connection header received
0619 16:50:19.939898 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.939903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.939958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.940008 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:19.940448 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.941028 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.942554 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943205 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.943305 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.943354 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.943679 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.943756 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:50:19.944459 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.939380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:54231 (local address 127.3.233.129:37971)
0619 16:50:19.939585 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.939589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.939712 (+ 123us) server_negotiation.cc:407] Connection header received
0619 16:50:19.940012 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.940016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.940069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.940110 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:19.940511 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.941235 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.943123 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943722 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.943811 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.943918 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.944266 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.944334 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":124}
I20260619 16:50:19.944461 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.939234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38767 (local address 127.3.233.130:52407)
0619 16:50:19.939578 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.939597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.939736 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.940159 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.940161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.940177 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:19.940302 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.940308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.941162 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.941166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.942429 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.942440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943446 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.943449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.943545 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.943565 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.944095 (+ 530us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.944290 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":99,"thread_start_us":88,"threads_started":1}
I20260619 16:50:19.945171 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.939233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:37971 (local address 127.3.233.130:54231)
0619 16:50:19.939589 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.939605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.939847 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.940220 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.940222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.940236 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:19.940359 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.940388 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.941363 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.941367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.942996 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.943006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943998 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.944002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.944114 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.944164 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.944627 (+ 463us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.944717 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":8,"thread_start_us":108,"threads_started":1}
I20260619 16:50:19.945487 22536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4482fdab06549f493a011451f29305d" candidate_uuid: "12dd338637c7437c9b274a571015f504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" is_pre_election: true
I20260619 16:50:19.945808 22536 raft_consensus.cc:2454] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12dd338637c7437c9b274a571015f504 in term 0.
I20260619 16:50:19.945955 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4482fdab06549f493a011451f29305d" candidate_uuid: "12dd338637c7437c9b274a571015f504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" is_pre_election: true
I20260619 16:50:19.946102 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.941909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:33501 (local address 127.3.233.130:39129)
0619 16:50:19.942012 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.942016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.942031 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:50:19.942072 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.942075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.942119 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.942207 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:19.942509 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943238 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.944767 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.945548 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.945640 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.945682 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.945934 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.946012 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:50:19.946162 22268 raft_consensus.cc:2454] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12dd338637c7437c9b274a571015f504 in term 0.
I20260619 16:50:19.946345 22337 leader_election.cc:302] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12dd338637c7437c9b274a571015f504, 7f1ca91524e84960ad9b6dd3817dd2ed; no voters:
I20260619 16:50:19.946529 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.941290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:38767 (local address 127.3.233.129:56601)
0619 16:50:19.942112 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.942126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.942215 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.942531 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.942537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.942555 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:19.942665 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.942671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943719 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.943722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.945109 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.945119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.946038 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.946044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.946188 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.946211 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.946293 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.946371 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":116,"threads_started":1}
I20260619 16:50:19.946678 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.941188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39129 (local address 127.3.233.129:33501)
0619 16:50:19.941685 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:19.941699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:19.941789 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:19.942233 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:19.942236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:19.942255 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:19.942393 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.942399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943407 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.943410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.944641 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:19.944650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.945653 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.945656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:19.945798 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.945819 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:50:19.946236 (+ 417us) client_negotiation.cc:240] Negotiation successful
0619 16:50:19.946313 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":108,"threads_started":1}
I20260619 16:50:19.946662 22606 raft_consensus.cc:2781] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:19.946758 22606 raft_consensus.cc:482] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:19.946812 22606 raft_consensus.cc:3037] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:19.946916 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.941684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:56601 (local address 127.3.233.131:38767)
0619 16:50:19.942029 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.942038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.942288 (+ 250us) server_negotiation.cc:407] Connection header received
0619 16:50:19.942332 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.942335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.942377 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.942448 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:19.942853 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.943601 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.945254 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.945823 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.945936 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.945979 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.946681 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.946764 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
I20260619 16:50:19.947237 22402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4482fdab06549f493a011451f29305d" candidate_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12dd338637c7437c9b274a571015f504" is_pre_election: true
I20260619 16:50:19.947429 22536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4482fdab06549f493a011451f29305d" candidate_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" is_pre_election: true
I20260619 16:50:19.948323 22606 raft_consensus.cc:504] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.948626 22402 raft_consensus.cc:2379] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc0de85fe3eb4853bb6b7e429dff6951 in current term 1: Already voted for candidate 12dd338637c7437c9b274a571015f504 in this term.
I20260619 16:50:19.948750 22606 leader_election.cc:288] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [CANDIDATE]: Term 1 election: Requested vote from peers fc0de85fe3eb4853bb6b7e429dff6951 (127.3.233.129:37971), 7f1ca91524e84960ad9b6dd3817dd2ed (127.3.233.131:38767)
I20260619 16:50:19.947587 22536 raft_consensus.cc:2454] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc0de85fe3eb4853bb6b7e429dff6951 in term 0.
I20260619 16:50:19.949159 22203 leader_election.cc:302] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f1ca91524e84960ad9b6dd3817dd2ed, fc0de85fe3eb4853bb6b7e429dff6951; no voters:
I20260619 16:50:19.949268 22268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4482fdab06549f493a011451f29305d" candidate_uuid: "12dd338637c7437c9b274a571015f504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0de85fe3eb4853bb6b7e429dff6951"
I20260619 16:50:19.949373 22268 raft_consensus.cc:3037] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:19.949361 22536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4482fdab06549f493a011451f29305d" candidate_uuid: "12dd338637c7437c9b274a571015f504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed"
I20260619 16:50:19.949473 22536 raft_consensus.cc:3037] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:19.950464 22268 raft_consensus.cc:2454] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12dd338637c7437c9b274a571015f504 in term 1.
I20260619 16:50:19.950551 22536 raft_consensus.cc:2454] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12dd338637c7437c9b274a571015f504 in term 1.
I20260619 16:50:19.950624 22605 raft_consensus.cc:2741] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 16:50:19.950917 22336 leader_election.cc:302] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12dd338637c7437c9b274a571015f504, fc0de85fe3eb4853bb6b7e429dff6951; no voters:
I20260619 16:50:19.951119 22606 raft_consensus.cc:2781] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:19.951346 22606 raft_consensus.cc:686] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 1 LEADER]: Becoming Leader. State: Replica: 12dd338637c7437c9b274a571015f504, State: Running, Role: LEADER
I20260619 16:50:19.951730 22606 consensus_queue.cc:237] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } }
I20260619 16:50:19.955111 22119 catalog_manager.cc:5495] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12dd338637c7437c9b274a571015f504 (127.3.233.130). New cstate: current_term: 1 leader_uuid: "12dd338637c7437c9b274a571015f504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12dd338637c7437c9b274a571015f504" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 } health_report { overall_health: UNKNOWN } } }
I20260619 16:50:19.968734 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:19.965123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.3.233.130:39129)
0619 16:50:19.965265 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:19.965269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:19.965286 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:50:19.965410 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:19.965413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:19.965469 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:19.965557 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:19.965793 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.966373 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.967350 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:19.967564 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:19.967646 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:19.967919 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:19.968324 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:19.968434 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:19.968589 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 16:50:19.968623 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:50:19.975107 22536 raft_consensus.cc:1261] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 1 FOLLOWER]: Refusing update from remote peer 12dd338637c7437c9b274a571015f504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:19.975107 22268 raft_consensus.cc:1261] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951 [term 1 FOLLOWER]: Refusing update from remote peer 12dd338637c7437c9b274a571015f504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:19.975821 22606 consensus_queue.cc:1035] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc0de85fe3eb4853bb6b7e429dff6951" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:19.975999 22613 consensus_queue.cc:1035] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f1ca91524e84960ad9b6dd3817dd2ed" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 38767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:19.983907 22617 mvcc.cc:205] Tried to move back new op lower bound from 7298612510610210816 to 7298612510524821504. Current Snapshot: MvccSnapshot[applied={T|T < 7298612510610210816}]
I20260619 16:50:19.993600 22123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:19.993798 22123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:19.996294 22123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 331e29e2a09d4b018cec44006a68e01d: Sending DeleteTablet for 3 replicas of tablet b4482fdab06549f493a011451f29305d
I20260619 16:50:19.997210 22248 tablet_service.cc:1514] Processing DeleteTablet for tablet b4482fdab06549f493a011451f29305d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:19 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:41826
I20260619 16:50:19.997283 22516 tablet_service.cc:1514] Processing DeleteTablet for tablet b4482fdab06549f493a011451f29305d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:19 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:52432
I20260619 16:50:19.997712 22382 tablet_service.cc:1514] Processing DeleteTablet for tablet b4482fdab06549f493a011451f29305d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:19 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:50576
I20260619 16:50:19.997989 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22185
I20260619 16:50:19.998180 22626 tablet_replica.cc:330] T b4482fdab06549f493a011451f29305d P fc0de85fe3eb4853bb6b7e429dff6951: stopping tablet replica
I20260619 16:50:19.998837 22627 tablet_replica.cc:330] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: stopping tablet replica
I20260619 16:50:19.999168 22627 raft_consensus.cc:2229] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:19.999492 22627 raft_consensus.cc:2258] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:20.001212 22627 ts_tablet_manager.cc:1905] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:20.002068 22628 tablet_replica.cc:330] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: stopping tablet replica
I20260619 16:50:20.002347 22628 raft_consensus.cc:2229] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:20.002812 22628 raft_consensus.cc:2258] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:20.004489 22628 ts_tablet_manager.cc:1905] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:20.004899 22627 ts_tablet_manager.cc:1918] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:20.004988 22627 log.cc:1196] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/b4482fdab06549f493a011451f29305d
I20260619 16:50:20.005290 22627 ts_tablet_manager.cc:1939] T b4482fdab06549f493a011451f29305d P 7f1ca91524e84960ad9b6dd3817dd2ed: Deleting consensus metadata
W20260619 16:50:20.006603 22108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:37971 (error 108)
I20260619 16:50:20.007270 22109 catalog_manager.cc:4841] TS 7f1ca91524e84960ad9b6dd3817dd2ed (127.3.233.131:38767): tablet b4482fdab06549f493a011451f29305d (table test-table [id=e5532d15ff924ff8830a62b752fb206e]) successfully deleted
I20260619 16:50:20.007587 22628 ts_tablet_manager.cc:1918] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:20.007663 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22319
I20260619 16:50:20.007696 22628 log.cc:1196] T b4482fdab06549f493a011451f29305d P 12dd338637c7437c9b274a571015f504: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/b4482fdab06549f493a011451f29305d
I20260619 16:50:20.009902 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.009594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:37971 (local address 127.0.0.1:41842)
0619 16:50:20.009758 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.009823 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:50:20.010076 22108 catalog_manager.cc:4584] TS fc0de85fe3eb4853bb6b7e429dff6951 (127.3.233.129:37971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4482fdab06549f493a011451f29305d: Network error: Client connection negotiation failed: client connection to 127.3.233.129:37971: connect: Connection refused (error 111)
I20260619 16:50:20.013985 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22453
I20260619 16:50:20.015849 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.014221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39129 (local address 127.0.0.1:50596)
0619 16:50:20.015654 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.015728 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.130:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1325}
I20260619 16:50:20.020033 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22091
2026-06-19T16:50:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2631 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T16:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:20Z Disabled control of system clock
I20260619 16:50:20.050299 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:44385
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:50:20.163435 22633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:20.163727 22633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:20.163793 22633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:20.167974 22633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:20.168040 22633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:20.168056 22633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:20.168073 22633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:20.168128 22633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:20.172993 22633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:44385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:20.174135 22633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:20.175662 22633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:20.184613 22638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:20.184613 22641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:20.184634 22639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:20.184890 22633 server_base.cc:1034] running on GCE node
I20260619 16:50:20.185683 22633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:20.186894 22633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:20.188122 22633 hybrid_clock.cc:648] HybridClock initialized: now 1781887820188087 us; error 53 us; skew 500 ppm
I20260619 16:50:20.190083 22633 webserver.cc:460] Webserver started at http://127.3.233.190:36953/ using document root <none> and password file <none>
I20260619 16:50:20.190690 22633 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:20.190753 22633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:20.191037 22633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:20.193153 22633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "84ba8f0fbf8342949545bfdc2627cddd"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "b4e1870d2f87021c1633a30a84b6a820"
server_key_iv: "446de9e369dc94b5bc881fab57f98530"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.193739 22633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "84ba8f0fbf8342949545bfdc2627cddd"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "b4e1870d2f87021c1633a30a84b6a820"
server_key_iv: "446de9e369dc94b5bc881fab57f98530"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.197719 22633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:50:20.200276 22647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.201514 22633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:20.201650 22633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "84ba8f0fbf8342949545bfdc2627cddd"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "b4e1870d2f87021c1633a30a84b6a820"
server_key_iv: "446de9e369dc94b5bc881fab57f98530"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.201767 22633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:20.216992 22633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:20.217747 22633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:20.217975 22633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:20.225198 22633 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:44385
I20260619 16:50:20.225272 22699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:44385 every 8 connection(s)
I20260619 16:50:20.226284 22633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:20.229485 22700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:20.234915 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd: Bootstrap starting.
I20260619 16:50:20.236544 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22633
I20260619 16:50:20.236709 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:20.237017 4006 external_mini_cluster.cc:1428] Setting key 9ecbad2705ad28363c198920ae9c820a
I20260619 16:50:20.237689 22700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:20.238561 22700 log.cc:824] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:20.240991 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd: No bootstrap required, opened a new log
I20260619 16:50:20.245456 22700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ba8f0fbf8342949545bfdc2627cddd" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44385 } }
I20260619 16:50:20.245699 22700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:20.245746 22700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ba8f0fbf8342949545bfdc2627cddd, State: Initialized, Role: FOLLOWER
I20260619 16:50:20.246325 22700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ba8f0fbf8342949545bfdc2627cddd" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44385 } }
I20260619 16:50:20.246495 22700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:20.246583 22700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:20.246685 22700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:20.247424 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.238364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.3.233.190:44385)
0619 16:50:20.239259 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.239276 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.239312 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:50:20.239993 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.240011 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.240419 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.240721 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.241118 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.242138 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.243443 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.243940 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.244074 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.245132 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:20.245138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:20.245152 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:20.245647 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:20.245764 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:20.245800 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:20.245928 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.246089 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.246399 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":322,"threads_started":1}
I20260619 16:50:20.247822 22700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ba8f0fbf8342949545bfdc2627cddd" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44385 } }
I20260619 16:50:20.248203 22700 leader_election.cc:302] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ba8f0fbf8342949545bfdc2627cddd; no voters:
I20260619 16:50:20.248533 22700 leader_election.cc:288] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:20.248714 22705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:20.248960 22705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [term 1 LEADER]: Becoming Leader. State: Replica: 84ba8f0fbf8342949545bfdc2627cddd, State: Running, Role: LEADER
I20260619 16:50:20.249341 22705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ba8f0fbf8342949545bfdc2627cddd" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44385 } }
I20260619 16:50:20.249912 22700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:20.251390 22707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ba8f0fbf8342949545bfdc2627cddd. Latest consensus state: current_term: 1 leader_uuid: "84ba8f0fbf8342949545bfdc2627cddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ba8f0fbf8342949545bfdc2627cddd" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44385 } } }
I20260619 16:50:20.251425 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ba8f0fbf8342949545bfdc2627cddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ba8f0fbf8342949545bfdc2627cddd" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44385 } } }
I20260619 16:50:20.251523 22707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:20.251549 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:20.252027 22718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:20.254904 22718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:20.260929 22718 catalog_manager.cc:1347] Generated new cluster ID: c6d6c9ab8e464ee0a2f5a11e895a233d
I20260619 16:50:20.261013 22718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:20.269388 22718 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:20.270659 22718 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:20.278551 22718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd: Generated new TSK 0
I20260619 16:50:20.279449 22718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:20.283799 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44385
--builtin_ntp_servers=127.3.233.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:50:20.411616 22724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:20.411870 22724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:20.411909 22724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:20.416081 22724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:20.416150 22724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:20.416209 22724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:50:20.421320 22724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:20.422498 22724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:20.424015 22724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:20.434216 22732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:20.434216 22730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:20.434304 22724 server_base.cc:1034] running on GCE node
W20260619 16:50:20.434243 22729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:20.435038 22724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:20.435799 22724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:20.437005 22724 hybrid_clock.cc:648] HybridClock initialized: now 1781887820436988 us; error 31 us; skew 500 ppm
I20260619 16:50:20.439098 22724 webserver.cc:460] Webserver started at http://127.3.233.129:45963/ using document root <none> and password file <none>
I20260619 16:50:20.439703 22724 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:20.439797 22724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:20.440040 22724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:20.442219 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "142bfddeb1db49ce867a18dd93946b03"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "f15c1ac88c5e89ce36d74adbe9e40a7e"
server_key_iv: "c13629fdbb5dc0592f7a26a762f1c587"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.442888 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "142bfddeb1db49ce867a18dd93946b03"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "f15c1ac88c5e89ce36d74adbe9e40a7e"
server_key_iv: "c13629fdbb5dc0592f7a26a762f1c587"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.446892 22724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 16:50:20.449985 22738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.451747 22724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:50:20.451924 22724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "142bfddeb1db49ce867a18dd93946b03"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "f15c1ac88c5e89ce36d74adbe9e40a7e"
server_key_iv: "c13629fdbb5dc0592f7a26a762f1c587"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.452065 22724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:20.466557 22724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:20.467444 22724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:20.467666 22724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:20.468310 22724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:20.469340 22724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:20.469413 22724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.469491 22724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:20.469533 22724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.478735 22724 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:46523
I20260619 16:50:20.478806 22851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:46523 every 8 connection(s)
I20260619 16:50:20.479786 22724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:20.481468 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22724
I20260619 16:50:20.481619 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:20.481957 4006 external_mini_cluster.cc:1428] Setting key db7630e2a674a3e41cfd60f1c3ce2054
I20260619 16:50:20.484246 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44385
--builtin_ntp_servers=127.3.233.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:50:20.492132 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.481953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:58015 (local address 127.3.233.190:44385)
0619 16:50:20.482106 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.482112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.483521 (+ 1409us) server_negotiation.cc:407] Connection header received
0619 16:50:20.485119 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.485126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.485198 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.485307 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.486788 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.487521 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.489520 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.489783 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.489884 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.491546 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:20.491549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:20.491551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:20.491565 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:20.491586 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:20.491595 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:20.491674 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.491938 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.492005 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:50:20.493526 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.482503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:44385 (local address 127.3.233.129:58015)
0619 16:50:20.483366 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.483425 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.484863 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.485602 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.485616 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.485765 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:20.486557 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.486583 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.487674 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.487681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.489365 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.489381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.489974 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.489980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.490248 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.491149 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:20.491153 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:20.491225 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:20.491352 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:20.491708 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:20.491722 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:20.491732 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.491895 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.492276 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":178,"threads_started":1}
I20260619 16:50:20.495167 22852 heartbeater.cc:347] Connected to a master server at 127.3.233.190:44385
I20260619 16:50:20.495500 22852 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:20.496094 22852 heartbeater.cc:511] Master 127.3.233.190:44385 requested a full tablet report, sending...
I20260619 16:50:20.497879 22664 ts_manager.cc:194] Registered new tserver with Master: 142bfddeb1db49ce867a18dd93946b03 (127.3.233.129:46523)
I20260619 16:50:20.499511 22664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.129:58015
W20260619 16:50:20.602031 22855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:20.602324 22855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:20.602365 22855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:20.606846 22855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:20.606916 22855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:20.607034 22855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:50:20.611944 22855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:20.613020 22855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:20.614341 22855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:20.624601 22861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:20.624681 22855 server_base.cc:1034] running on GCE node
W20260619 16:50:20.624605 22860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:20.624619 22863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:20.625392 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:20.626152 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:20.627357 22855 hybrid_clock.cc:648] HybridClock initialized: now 1781887820627319 us; error 53 us; skew 500 ppm
I20260619 16:50:20.629540 22855 webserver.cc:460] Webserver started at http://127.3.233.130:41277/ using document root <none> and password file <none>
I20260619 16:50:20.630215 22855 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:20.630314 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:20.630525 22855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:20.632690 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "b1627869f9144e1c92d66289fd04bc58"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "509a2b568db87e18193524dde1143f50"
server_key_iv: "2aefa8fecddb04df7289b89c8e837abd"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.633299 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "b1627869f9144e1c92d66289fd04bc58"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "509a2b568db87e18193524dde1143f50"
server_key_iv: "2aefa8fecddb04df7289b89c8e837abd"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.637229 22855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 16:50:20.639848 22869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.641170 22855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:20.641301 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "b1627869f9144e1c92d66289fd04bc58"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "509a2b568db87e18193524dde1143f50"
server_key_iv: "2aefa8fecddb04df7289b89c8e837abd"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.641418 22855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:20.659953 22855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:20.660706 22855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:20.660934 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:20.661566 22855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:20.662555 22855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:20.662629 22855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.662699 22855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:20.662757 22855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.671895 22855 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:39465
I20260619 16:50:20.671936 22982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:39465 every 8 connection(s)
I20260619 16:50:20.672863 22855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:20.681741 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22855
I20260619 16:50:20.681842 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:20.682152 4006 external_mini_cluster.cc:1428] Setting key 7ab0017ca7925432331f0ef7cb3e157a
I20260619 16:50:20.682742 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.674975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:58481 (local address 127.3.233.190:44385)
0619 16:50:20.675104 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.675108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.676267 (+ 1159us) server_negotiation.cc:407] Connection header received
0619 16:50:20.677703 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.677706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.677758 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.677842 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.678787 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.679406 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.680537 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.680757 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.680840 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.682162 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:20.682166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:20.682169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:20.682182 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:20.682210 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:20.682217 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:20.682368 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.682534 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.682598 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:50:20.683866 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.675398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:44385 (local address 127.3.233.130:58481)
0619 16:50:20.676093 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.676135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.677480 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.678002 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.678008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.678095 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:20.678647 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.678659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.679512 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.679514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.680415 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.680423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.680924 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.680927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.681103 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.681868 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:20.681871 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:20.681933 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:20.682003 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:20.682368 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:20.682379 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:20.682388 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.682533 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.682962 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
I20260619 16:50:20.684412 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44385
--builtin_ntp_servers=127.3.233.148:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:50:20.685119 22983 heartbeater.cc:347] Connected to a master server at 127.3.233.190:44385
I20260619 16:50:20.685426 22983 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:20.685967 22983 heartbeater.cc:511] Master 127.3.233.190:44385 requested a full tablet report, sending...
I20260619 16:50:20.687129 22664 ts_manager.cc:194] Registered new tserver with Master: b1627869f9144e1c92d66289fd04bc58 (127.3.233.130:39465)
I20260619 16:50:20.687978 22664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.130:58481
W20260619 16:50:20.798843 22986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:20.799160 22986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:20.799201 22986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:20.803295 22986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:20.803362 22986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:20.803421 22986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:50:20.808437 22986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:20.809556 22986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:20.811156 22986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:20.821064 22994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:20.821024 22991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:20.821024 22992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:20.821174 22986 server_base.cc:1034] running on GCE node
I20260619 16:50:20.821718 22986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:20.822381 22986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:20.823581 22986 hybrid_clock.cc:648] HybridClock initialized: now 1781887820823563 us; error 29 us; skew 500 ppm
I20260619 16:50:20.825419 22986 webserver.cc:460] Webserver started at http://127.3.233.131:34181/ using document root <none> and password file <none>
I20260619 16:50:20.826009 22986 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:20.826104 22986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:20.826350 22986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:20.828461 22986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "9b7550b4b1f848f3ba11a55f87d559da"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "21e18792737119e3d2d7569767f0f9fa"
server_key_iv: "be9395376150f6d9112914640f8872f8"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.829015 22986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "9b7550b4b1f848f3ba11a55f87d559da"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "21e18792737119e3d2d7569767f0f9fa"
server_key_iv: "be9395376150f6d9112914640f8872f8"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.832990 22986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:20.835565 23000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.837016 22986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:20.837163 22986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "9b7550b4b1f848f3ba11a55f87d559da"
format_stamp: "Formatted at 2026-06-19 16:50:20 on dist-test-slave-cgwx"
server_key: "21e18792737119e3d2d7569767f0f9fa"
server_key_iv: "be9395376150f6d9112914640f8872f8"
server_key_version: "encryptionkey@0"
I20260619 16:50:20.837293 22986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:20.863297 22986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:20.864081 22986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:20.864305 22986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:20.864941 22986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:20.865955 22986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:20.866032 22986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.866110 22986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:20.866156 22986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:20.875738 22986 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:46751
I20260619 16:50:20.875758 23113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:46751 every 8 connection(s)
I20260619 16:50:20.876736 22986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:20.882005 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 22986
I20260619 16:50:20.882159 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:20.882498 4006 external_mini_cluster.cc:1428] Setting key 0bcbadb8595b33c9f8fd7cbd4ddad3d0
I20260619 16:50:20.887737 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.878787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:56581 (local address 127.3.233.190:44385)
0619 16:50:20.878983 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.878994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.880133 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 16:50:20.881821 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.881825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.881896 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.882011 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.883269 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.884061 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.885689 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.885917 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.886009 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.887282 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:20.887286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:20.887288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:20.887300 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:20.887319 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:20.887326 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:20.887394 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.887563 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.887615 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:50:20.888661 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.879216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:44385 (local address 127.3.233.131:56581)
0619 16:50:20.879959 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.880006 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.881541 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.882268 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.882281 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.882392 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:20.883116 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.883143 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.884272 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.884276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.885545 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.885556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.886090 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.886093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.886290 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.886898 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:20.886899 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:20.887002 (+ 103us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:20.887083 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:20.887421 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:20.887428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:20.887433 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.887531 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.887775 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260619 16:50:20.889801 23114 heartbeater.cc:347] Connected to a master server at 127.3.233.190:44385
I20260619 16:50:20.890023 23114 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:20.890548 23114 heartbeater.cc:511] Master 127.3.233.190:44385 requested a full tablet report, sending...
I20260619 16:50:20.891592 22664 ts_manager.cc:194] Registered new tserver with Master: 9b7550b4b1f848f3ba11a55f87d559da (127.3.233.131:46751)
I20260619 16:50:20.892129 22664 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.131:56581
I20260619 16:50:20.896922 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:20.903285 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.899052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.3.233.190:44385)
0619 16:50:20.899212 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.899217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.899405 (+ 188us) server_negotiation.cc:407] Connection header received
0619 16:50:20.899574 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.899577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.899630 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.899715 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:20.900007 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.900529 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.901387 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.901635 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.901750 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.902730 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:20.902734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:20.902738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:20.902756 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:20.902774 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:20.902782 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:20.902951 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.903076 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.903128 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:50:20.906508 22664 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:20.909633 22664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:20.924969 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.919395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:46751 (local address 127.0.0.1:59680)
0619 16:50:20.920333 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.920346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.920483 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.920967 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.920970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.920988 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:20.921063 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.921068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.922335 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.922338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.923493 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.923505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.924411 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.924415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.924520 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.924556 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.924669 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.924804 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":23,"thread_start_us":85,"threads_started":1}
I20260619 16:50:20.924993 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.919775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.3.233.131:46751)
0619 16:50:20.920167 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.920180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.920417 (+ 237us) server_negotiation.cc:407] Connection header received
0619 16:50:20.920633 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.920638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.920809 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.920948 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:20.921169 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.922173 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.923665 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.924287 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.924408 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.924608 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.924720 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.924814 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":86,"threads_started":1}
I20260619 16:50:20.925201 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.918572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39465 (local address 127.0.0.1:52948)
0619 16:50:20.919395 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.919451 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.919584 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.920346 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.920352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.920374 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:20.920513 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.920520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.921917 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.921920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.923312 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.923322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.924675 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.924679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.924839 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.924923 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.924986 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.925064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":187,"thread_start_us":88,"threads_started":1}
I20260619 16:50:20.925299 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.919134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.3.233.129:46523)
0619 16:50:20.919510 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.919525 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.919971 (+ 446us) server_negotiation.cc:407] Connection header received
0619 16:50:20.920243 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.920249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.920448 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.920569 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:20.920947 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.921752 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.923624 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.924354 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.924456 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.924609 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.925046 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.925143 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":149,"threads_started":1}
I20260619 16:50:20.925630 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.918791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:46523 (local address 127.0.0.1:59280)
0619 16:50:20.919866 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.919882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.920052 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.920730 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.920732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.920746 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:20.920823 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.920827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.921867 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.921874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.923493 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.923505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.924675 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.924679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.924834 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.924885 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.925437 (+ 552us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.925506 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":124,"thread_start_us":112,"threads_started":1}
I20260619 16:50:20.926182 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.918743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.3.233.130:39465)
0619 16:50:20.919804 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.919815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.919840 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:50:20.919917 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.919923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.920160 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.920330 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:20.920649 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.921817 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.923448 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.924290 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.924415 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.924609 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.925372 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.925474 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":849,"threads_started":1}
I20260619 16:50:20.928596 22917 tablet_service.cc:1467] Processing CreateTablet for tablet 77c2f5841b824cd39b9581a90c000d75 (DEFAULT_TABLE table=test-table [id=ec45faa10ad84cff972fc93bb52d7a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:20.928632 23048 tablet_service.cc:1467] Processing CreateTablet for tablet 77c2f5841b824cd39b9581a90c000d75 (DEFAULT_TABLE table=test-table [id=ec45faa10ad84cff972fc93bb52d7a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:20.928771 22786 tablet_service.cc:1467] Processing CreateTablet for tablet 77c2f5841b824cd39b9581a90c000d75 (DEFAULT_TABLE table=test-table [id=ec45faa10ad84cff972fc93bb52d7a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:20.929620 22917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c2f5841b824cd39b9581a90c000d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:20.929677 23048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c2f5841b824cd39b9581a90c000d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:20.929677 22786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c2f5841b824cd39b9581a90c000d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:20.936352 23131 tablet_bootstrap.cc:492] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: Bootstrap starting.
I20260619 16:50:20.937810 23132 tablet_bootstrap.cc:492] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03: Bootstrap starting.
I20260619 16:50:20.939162 23131 tablet_bootstrap.cc:654] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:20.939281 23133 tablet_bootstrap.cc:492] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Bootstrap starting.
I20260619 16:50:20.940016 23132 tablet_bootstrap.cc:654] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:20.940078 23131 log.cc:824] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:20.941012 23132 log.cc:824] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:20.941316 23133 tablet_bootstrap.cc:654] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:20.942173 23133 log.cc:824] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:20.942732 23131 tablet_bootstrap.cc:492] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: No bootstrap required, opened a new log
I20260619 16:50:20.943017 23131 ts_tablet_manager.cc:1397] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:50:20.943399 23132 tablet_bootstrap.cc:492] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03: No bootstrap required, opened a new log
I20260619 16:50:20.943611 23132 ts_tablet_manager.cc:1397] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:50:20.944176 23133 tablet_bootstrap.cc:492] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: No bootstrap required, opened a new log
I20260619 16:50:20.944356 23133 ts_tablet_manager.cc:1397] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:50:20.947338 23132 raft_consensus.cc:348] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.947603 23132 raft_consensus.cc:374] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:20.947659 23132 raft_consensus.cc:729] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142bfddeb1db49ce867a18dd93946b03, State: Initialized, Role: FOLLOWER
I20260619 16:50:20.947605 23131 raft_consensus.cc:348] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.947890 23131 raft_consensus.cc:374] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:20.947942 23131 raft_consensus.cc:729] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1627869f9144e1c92d66289fd04bc58, State: Initialized, Role: FOLLOWER
I20260619 16:50:20.948155 23132 consensus_queue.cc:260] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.948367 23131 consensus_queue.cc:260] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.948601 23133 raft_consensus.cc:348] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.948839 23133 raft_consensus.cc:374] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:20.948902 23133 raft_consensus.cc:729] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7550b4b1f848f3ba11a55f87d559da, State: Initialized, Role: FOLLOWER
I20260619 16:50:20.949359 23131 ts_tablet_manager.cc:1428] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:50:20.949311 23133 consensus_queue.cc:260] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.949572 23132 ts_tablet_manager.cc:1428] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:50:20.949777 22983 heartbeater.cc:503] Master 127.3.233.190:44385 was elected leader, sending a full tablet report...
I20260619 16:50:20.949924 22852 heartbeater.cc:503] Master 127.3.233.190:44385 was elected leader, sending a full tablet report...
I20260619 16:50:20.949998 23114 heartbeater.cc:503] Master 127.3.233.190:44385 was elected leader, sending a full tablet report...
I20260619 16:50:20.950191 23133 ts_tablet_manager.cc:1428] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:50:20.957563 23138 raft_consensus.cc:482] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:20.957770 23138 raft_consensus.cc:504] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.959722 23138 leader_election.cc:288] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b7550b4b1f848f3ba11a55f87d559da (127.3.233.131:46751), 142bfddeb1db49ce867a18dd93946b03 (127.3.233.129:46523)
I20260619 16:50:20.963866 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.959093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:46751 (local address 127.3.233.130:35373)
0619 16:50:20.959393 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.959413 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.959505 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.960443 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.960446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.960462 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:20.960590 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.960598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.961659 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.961664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.962761 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.962768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.963439 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.963443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.963542 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.963563 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.963659 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.963740 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 16:50:20.963866 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.959284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:35373 (local address 127.3.233.131:46751)
0619 16:50:20.960212 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.960216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.960239 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:50:20.960298 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.960301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.960351 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.960432 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:20.960686 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.961564 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.962873 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.963319 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.963451 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.963500 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.963672 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.963739 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822}
I20260619 16:50:20.964473 23068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c2f5841b824cd39b9581a90c000d75" candidate_uuid: "b1627869f9144e1c92d66289fd04bc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b7550b4b1f848f3ba11a55f87d559da" is_pre_election: true
I20260619 16:50:20.964785 23068 raft_consensus.cc:2454] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1627869f9144e1c92d66289fd04bc58 in term 0.
I20260619 16:50:20.965266 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.960733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:49711 (local address 127.3.233.129:46523)
0619 16:50:20.961996 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.962002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.962018 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:20.962074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.962077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.962125 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.962231 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:20.962510 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.963047 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.964106 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.964611 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.964706 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.964739 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.965090 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.965163 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1165}
I20260619 16:50:20.965262 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.959507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:46523 (local address 127.3.233.130:49711)
0619 16:50:20.959894 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:20.959907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:20.959989 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:20.962270 (+ 2281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:20.962274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:20.962292 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:20.962406 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.962411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.963154 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.963158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.963946 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:20.963959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.964835 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.964838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:20.964966 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.964983 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:50:20.965061 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:50:20.965127 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":103,"threads_started":1}
I20260619 16:50:20.965256 22872 leader_election.cc:302] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b7550b4b1f848f3ba11a55f87d559da, b1627869f9144e1c92d66289fd04bc58; no voters:
I20260619 16:50:20.965580 23138 raft_consensus.cc:2781] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:20.965665 23138 raft_consensus.cc:482] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:20.965724 23138 raft_consensus.cc:3037] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:20.965921 22806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c2f5841b824cd39b9581a90c000d75" candidate_uuid: "b1627869f9144e1c92d66289fd04bc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142bfddeb1db49ce867a18dd93946b03" is_pre_election: true
I20260619 16:50:20.966239 22806 raft_consensus.cc:2454] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1627869f9144e1c92d66289fd04bc58 in term 0.
I20260619 16:50:20.967005 23138 raft_consensus.cc:504] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.967350 23138 leader_election.cc:288] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [CANDIDATE]: Term 1 election: Requested vote from peers 9b7550b4b1f848f3ba11a55f87d559da (127.3.233.131:46751), 142bfddeb1db49ce867a18dd93946b03 (127.3.233.129:46523)
I20260619 16:50:20.967675 22806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c2f5841b824cd39b9581a90c000d75" candidate_uuid: "b1627869f9144e1c92d66289fd04bc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142bfddeb1db49ce867a18dd93946b03"
I20260619 16:50:20.967695 23068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c2f5841b824cd39b9581a90c000d75" candidate_uuid: "b1627869f9144e1c92d66289fd04bc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b7550b4b1f848f3ba11a55f87d559da"
I20260619 16:50:20.967819 23068 raft_consensus.cc:3037] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:20.967815 22806 raft_consensus.cc:3037] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:20.968914 22806 raft_consensus.cc:2454] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1627869f9144e1c92d66289fd04bc58 in term 1.
I20260619 16:50:20.968971 23068 raft_consensus.cc:2454] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1627869f9144e1c92d66289fd04bc58 in term 1.
I20260619 16:50:20.969226 22870 leader_election.cc:302] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142bfddeb1db49ce867a18dd93946b03, b1627869f9144e1c92d66289fd04bc58; no voters:
I20260619 16:50:20.969462 23138 raft_consensus.cc:2781] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:20.969679 23138 raft_consensus.cc:686] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 1 LEADER]: Becoming Leader. State: Replica: b1627869f9144e1c92d66289fd04bc58, State: Running, Role: LEADER
I20260619 16:50:20.970009 23138 consensus_queue.cc:237] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } }
I20260619 16:50:20.973321 22664 catalog_manager.cc:5495] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1627869f9144e1c92d66289fd04bc58 (127.3.233.130). New cstate: current_term: 1 leader_uuid: "b1627869f9144e1c92d66289fd04bc58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1627869f9144e1c92d66289fd04bc58" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 39465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 } health_report { overall_health: UNKNOWN } } }
W20260619 16:50:20.981025 22853 tablet.cc:2369] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:20.990357 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:20.986227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.3.233.130:39465)
0619 16:50:20.986392 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:20.986396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:20.986408 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:50:20.986471 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:20.986473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:20.986525 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:20.986642 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:20.986906 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.987434 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.988788 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:20.989020 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:20.989131 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:20.989500 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:20.989917 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:20.990080 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:20.990168 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 16:50:20.990205 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:50:20.996600 23068 raft_consensus.cc:1261] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 1 FOLLOWER]: Refusing update from remote peer b1627869f9144e1c92d66289fd04bc58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:20.996601 22806 raft_consensus.cc:1261] T 77c2f5841b824cd39b9581a90c000d75 P 142bfddeb1db49ce867a18dd93946b03 [term 1 FOLLOWER]: Refusing update from remote peer b1627869f9144e1c92d66289fd04bc58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:20.997426 23138 consensus_queue.cc:1035] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142bfddeb1db49ce867a18dd93946b03" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:20.997690 23141 consensus_queue.cc:1035] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b7550b4b1f848f3ba11a55f87d559da" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 46751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:21.005138 23144 mvcc.cc:205] Tried to move back new op lower bound from 7298612514793480192 to 7298612514695540736. Current Snapshot: MvccSnapshot[applied={T|T < 7298612514793480192}]
I20260619 16:50:21.006165 23145 mvcc.cc:205] Tried to move back new op lower bound from 7298612514793480192 to 7298612514695540736. Current Snapshot: MvccSnapshot[applied={T|T < 7298612514793480192}]
I20260619 16:50:21.013934 22662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:21.014128 22662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:21.016636 22662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84ba8f0fbf8342949545bfdc2627cddd: Sending DeleteTablet for 3 replicas of tablet 77c2f5841b824cd39b9581a90c000d75
I20260619 16:50:21.017630 22917 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c2f5841b824cd39b9581a90c000d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:21 UTC) from {username='slave'} at 127.0.0.1:52948
I20260619 16:50:21.017669 22786 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c2f5841b824cd39b9581a90c000d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:21 UTC) from {username='slave'} at 127.0.0.1:59280
I20260619 16:50:21.017755 23048 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c2f5841b824cd39b9581a90c000d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:21 UTC) from {username='slave'} at 127.0.0.1:59680
I20260619 16:50:21.018258 23155 tablet_replica.cc:330] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: stopping tablet replica
I20260619 16:50:21.018497 23154 tablet_replica.cc:330] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: stopping tablet replica
I20260619 16:50:21.018589 23155 raft_consensus.cc:2229] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:21.018775 23154 raft_consensus.cc:2229] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:21.018949 23155 raft_consensus.cc:2258] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:21.019070 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22724
I20260619 16:50:21.019181 23154 raft_consensus.cc:2258] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:21.022560 23154 ts_tablet_manager.cc:1905] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:50:21.024863 22648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:46523 (error 108)
I20260619 16:50:21.025389 23154 ts_tablet_manager.cc:1918] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:21.025490 23154 log.cc:1196] T 77c2f5841b824cd39b9581a90c000d75 P b1627869f9144e1c92d66289fd04bc58: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/77c2f5841b824cd39b9581a90c000d75
I20260619 16:50:21.025688 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22855
I20260619 16:50:21.027962 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:21.027603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:46523 (local address 127.0.0.1:59294)
0619 16:50:21.027803 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:21.027884 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:46523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 16:50:21.027941 23155 ts_tablet_manager.cc:1905] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:50:21.028177 22648 catalog_manager.cc:4584] TS 142bfddeb1db49ce867a18dd93946b03 (127.3.233.129:46523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77c2f5841b824cd39b9581a90c000d75: Network error: Client connection negotiation failed: client connection to 127.3.233.129:46523: connect: Connection refused (error 111)
I20260619 16:50:21.031600 23155 ts_tablet_manager.cc:1918] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:21.031734 23155 log.cc:1196] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/77c2f5841b824cd39b9581a90c000d75
I20260619 16:50:21.032068 23155 ts_tablet_manager.cc:1939] T 77c2f5841b824cd39b9581a90c000d75 P 9b7550b4b1f848f3ba11a55f87d559da: Deleting consensus metadata
I20260619 16:50:21.033593 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22986
I20260619 16:50:21.033782 22650 catalog_manager.cc:4841] TS 9b7550b4b1f848f3ba11a55f87d559da (127.3.233.131:46751): tablet 77c2f5841b824cd39b9581a90c000d75 (table test-table [id=ec45faa10ad84cff972fc93bb52d7a9b]) successfully deleted
I20260619 16:50:21.034268 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:21.033608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:39465 (local address 127.0.0.1:52962)
0619 16:50:21.033732 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:21.033790 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.130:39465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:50:21.039414 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 22633
2026-06-19T16:50:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1018 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:21 dist-test-slave-cgwx krb5kdc[23159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:21 dist-test-slave-cgwx krb5kdc[23159](info): set up 2 sockets
Jun 19 16:50:21 dist-test-slave-cgwx krb5kdc[23159](info): commencing operation
krb5kdc: starting...
W20260619 16:50:22.083091 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:22 dist-test-slave-cgwx krb5kdc[23159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:22.206095 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37053
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:37053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:22.317585 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:22.317904 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:22.317979 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:22.321988 23172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:22.322085 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:22.322124 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:22.322161 23172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:22.322196 23172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:22.327497 23172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:37053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:22.328625 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:22.329705 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:22.338361 23178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:22.338392 23172 server_base.cc:1034] running on GCE node
W20260619 16:50:22.338407 23177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:22.338547 23180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:22.339133 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:22.340248 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:22.341442 23172 hybrid_clock.cc:648] HybridClock initialized: now 1781887822341410 us; error 45 us; skew 500 ppm
Jun 19 16:50:22 dist-test-slave-cgwx krb5kdc[23159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887822, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:22.344322 23172 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:22.345209 23172 webserver.cc:460] Webserver started at http://127.3.233.190:36553/ using document root <none> and password file <none>
I20260619 16:50:22.345803 23172 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:22.345897 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:22.346122 23172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:22.348224 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "5244f19f669c44a1b6ed2e325b049c0e"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "e857c926807c63f4a28243394ca30c2c"
server_key_iv: "77d514a465e22ba6f6f91da5cb9751ce"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.348795 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "5244f19f669c44a1b6ed2e325b049c0e"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "e857c926807c63f4a28243394ca30c2c"
server_key_iv: "77d514a465e22ba6f6f91da5cb9751ce"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.352700 23172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:22.355345 23187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:22.356583 23172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:50:22.356752 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "5244f19f669c44a1b6ed2e325b049c0e"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "e857c926807c63f4a28243394ca30c2c"
server_key_iv: "77d514a465e22ba6f6f91da5cb9751ce"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.356874 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:22.374155 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:22.377383 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:22.377544 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:22.384689 23172 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:37053
I20260619 16:50:22.384735 23239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:37053 every 8 connection(s)
I20260619 16:50:22.385653 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:22.388520 23240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:22.392946 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 23172
I20260619 16:50:22.393100 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:22.393395 4006 external_mini_cluster.cc:1428] Setting key c27de30caa5649de88a8691366892606
I20260619 16:50:22.394397 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e: Bootstrap starting.
I20260619 16:50:22.396785 23240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:22.397527 23240 log.cc:824] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:22.399406 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e: No bootstrap required, opened a new log
Jun 19 16:50:22 dist-test-slave-cgwx krb5kdc[23159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:22.403290 23240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f19f669c44a1b6ed2e325b049c0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37053 } }
I20260619 16:50:22.403600 23240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:22.403669 23240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5244f19f669c44a1b6ed2e325b049c0e, State: Initialized, Role: FOLLOWER
I20260619 16:50:22.404220 23240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f19f669c44a1b6ed2e325b049c0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37053 } }
I20260619 16:50:22.404398 23240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:22.404469 23240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:22.404565 23240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:22.405697 23240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f19f669c44a1b6ed2e325b049c0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37053 } }
I20260619 16:50:22.406112 23240 leader_election.cc:302] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5244f19f669c44a1b6ed2e325b049c0e; no voters:
I20260619 16:50:22.406491 23240 leader_election.cc:288] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:22.406611 23245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:22.406860 23245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [term 1 LEADER]: Becoming Leader. State: Replica: 5244f19f669c44a1b6ed2e325b049c0e, State: Running, Role: LEADER
I20260619 16:50:22.407223 23245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f19f669c44a1b6ed2e325b049c0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37053 } }
I20260619 16:50:22.407878 23240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:22.408466 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5244f19f669c44a1b6ed2e325b049c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f19f669c44a1b6ed2e325b049c0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37053 } } }
I20260619 16:50:22.408582 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:22.408668 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.394886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.3.233.190:37053)
0619 16:50:22.395342 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:22.395357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:22.395418 (+ 61us) server_negotiation.cc:407] Connection header received
0619 16:50:22.396079 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:22.396096 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:22.396445 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:22.396798 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:22.397649 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.398647 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.399626 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.400296 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.400474 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.402949 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:22.402956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:22.402973 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:22.403004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:22.405093 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:22.405519 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:22.405528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:22.405538 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:22.405689 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:22.406098 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:22.406101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:22.406103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:22.406690 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:22.406891 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:22.407262 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 16:50:22.407625 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260619 16:50:22.409164 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5244f19f669c44a1b6ed2e325b049c0e. Latest consensus state: current_term: 1 leader_uuid: "5244f19f669c44a1b6ed2e325b049c0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f19f669c44a1b6ed2e325b049c0e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37053 } } }
I20260619 16:50:22.409260 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:22.410115 23252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:22.413470 23252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:22.419446 23252 catalog_manager.cc:1347] Generated new cluster ID: 53095bae33864606a0ba9321b6993e41
I20260619 16:50:22.419559 23252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:22.432531 23252 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:22.433658 23252 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:22.444329 23252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e: Generated new TSK 0
I20260619 16:50:22.445098 23252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.233.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
Entry for principal kudu/127.3.233.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab.
I20260619 16:50:22.482162 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37053
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:22.596778 23266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:22.597078 23266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:22.597122 23266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:22.601241 23266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:22.601308 23266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:22.601374 23266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:50:22.607607 23266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.129.keytab
--principal=kudu/127.3.233.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:22.608834 23266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:22.610076 23266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:22.620154 23271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:22.620154 23274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:22.620227 23272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:22.620425 23266 server_base.cc:1034] running on GCE node
I20260619 16:50:22.620857 23266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:22.621600 23266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:22.622817 23266 hybrid_clock.cc:648] HybridClock initialized: now 1781887822622792 us; error 39 us; skew 500 ppm
Jun 19 16:50:22 dist-test-slave-cgwx krb5kdc[23159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887822, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:22.625954 23266 init.cc:377] Logged in from keytab as kudu/127.3.233.129@KRBTEST.COM (short username kudu)
I20260619 16:50:22.627002 23266 webserver.cc:460] Webserver started at http://127.3.233.129:42099/ using document root <none> and password file <none>
I20260619 16:50:22.627588 23266 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:22.627679 23266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:22.627918 23266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:22.630052 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "25477e4e06e04ebf998354ec285f23df"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "7541cbfa18bb9996aa61b13c73df1636"
server_key_iv: "067dcaf4c4cdeb5f270e6fb57a897d1b"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.630558 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "25477e4e06e04ebf998354ec285f23df"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "7541cbfa18bb9996aa61b13c73df1636"
server_key_iv: "067dcaf4c4cdeb5f270e6fb57a897d1b"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.634465 23266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:50:22.637245 23281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:22.638792 23266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:22.638904 23266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "25477e4e06e04ebf998354ec285f23df"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "7541cbfa18bb9996aa61b13c73df1636"
server_key_iv: "067dcaf4c4cdeb5f270e6fb57a897d1b"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.639099 23266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:22.666349 23266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:22.669360 23266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:22.669512 23266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:22.670045 23266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:22.670822 23266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:22.670874 23266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:22.670913 23266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:22.671005 23266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:22.680289 23266 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:41257
I20260619 16:50:22.680330 23394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:41257 every 8 connection(s)
I20260619 16:50:22.681191 23266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:22.689167 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 23266
I20260619 16:50:22.689392 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:22.689775 4006 external_mini_cluster.cc:1428] Setting key 5f6be1d03291b3bc804b9b1659f53c1c
Jun 19 16:50:22 dist-test-slave-cgwx krb5kdc[23159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887822, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.129@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:22.695932 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.683052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:36481 (local address 127.3.233.190:37053)
0619 16:50:22.683200 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:22.683206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:22.684122 (+ 916us) server_negotiation.cc:407] Connection header received
0619 16:50:22.685149 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:22.685152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:22.685207 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:22.685294 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:22.686483 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.687085 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.688241 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.688474 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.688585 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.691571 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:22.691575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:22.691577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:22.691599 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:22.693427 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:22.693946 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:22.693952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:22.693957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:22.694052 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:22.694395 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:22.694398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:22.694400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:22.694659 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:22.694770 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:22.695563 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 16:50:22.695717 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:50:22.696799 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.683495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:37053 (local address 127.3.233.129:36481)
0619 16:50:22.683964 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:22.684007 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:22.684940 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:22.685468 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:22.685475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:22.685846 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:22.686333 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.686346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.687219 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.687222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.688086 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.688095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.688627 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.688631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.688811 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.689454 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:22.689477 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:22.691407 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:22.693551 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:22.693556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:22.693569 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:22.693836 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:22.694149 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:22.694151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:22.694153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:22.694290 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:22.694806 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:22.694813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:22.695169 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 16:50:22.695486 (+ 317us) client_negotiation.cc:240] Negotiation successful
0619 16:50:22.695805 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260619 16:50:22.698133 23395 heartbeater.cc:347] Connected to a master server at 127.3.233.190:37053
I20260619 16:50:22.698414 23395 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:22.699087 23395 heartbeater.cc:511] Master 127.3.233.190:37053 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.233.130@KRBTEST.COM; defaulting to no policy
I20260619 16:50:22.701037 23204 ts_manager.cc:194] Registered new tserver with Master: 25477e4e06e04ebf998354ec285f23df (127.3.233.129:41257)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.130@KRBTEST.COM" created.
I20260619 16:50:22.702327 23204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.129@KRBTEST.COM'} at 127.3.233.129:36481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
Entry for principal kudu/127.3.233.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab.
I20260619 16:50:22.717778 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37053
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:22.832330 23400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:22.832623 23400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:22.832674 23400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:22.837376 23400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:22.837450 23400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:22.837559 23400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:50:22.843613 23400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.130.keytab
--principal=kudu/127.3.233.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:22.844792 23400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:22.845919 23400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:22.856158 23406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:22.856119 23408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:22.856098 23405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:22.856599 23400 server_base.cc:1034] running on GCE node
I20260619 16:50:22.857019 23400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:22.857712 23400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:22.858912 23400 hybrid_clock.cc:648] HybridClock initialized: now 1781887822858895 us; error 30 us; skew 500 ppm
Jun 19 16:50:22 dist-test-slave-cgwx krb5kdc[23159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887822, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:22.862308 23400 init.cc:377] Logged in from keytab as kudu/127.3.233.130@KRBTEST.COM (short username kudu)
I20260619 16:50:22.863277 23400 webserver.cc:460] Webserver started at http://127.3.233.130:40367/ using document root <none> and password file <none>
I20260619 16:50:22.863852 23400 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:22.863914 23400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:22.864077 23400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:22.866060 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "f366098573dc4b2fa4a6de45f490beaa"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "640edff7bfb66e8e22eb5ca3b7809b27"
server_key_iv: "c05f2cba8bccdd66b2f2ab65887d0272"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.866580 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "f366098573dc4b2fa4a6de45f490beaa"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "640edff7bfb66e8e22eb5ca3b7809b27"
server_key_iv: "c05f2cba8bccdd66b2f2ab65887d0272"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.870244 23400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:50:22.872776 23415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:22.874050 23400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:50:22.874182 23400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "f366098573dc4b2fa4a6de45f490beaa"
format_stamp: "Formatted at 2026-06-19 16:50:22 on dist-test-slave-cgwx"
server_key: "640edff7bfb66e8e22eb5ca3b7809b27"
server_key_iv: "c05f2cba8bccdd66b2f2ab65887d0272"
server_key_version: "encryptionkey@0"
I20260619 16:50:22.874291 23400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:22.901660 23400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:22.905130 23400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:22.905287 23400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:22.905941 23400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:22.906790 23400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:22.906836 23400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:22.906877 23400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:22.906951 23400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:22.916126 23400 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:46209
I20260619 16:50:22.916149 23528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:46209 every 8 connection(s)
I20260619 16:50:22.917016 23400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:22.924610 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 23400
I20260619 16:50:22.924718 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:22.925141 4006 external_mini_cluster.cc:1428] Setting key 4e24f5dd959c44a408c176899daab10d
Jun 19 16:50:22 dist-test-slave-cgwx krb5kdc[23159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887822, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.130@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:22.931061 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.919007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:57905 (local address 127.3.233.190:37053)
0619 16:50:22.919143 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:22.919147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:22.920003 (+ 856us) server_negotiation.cc:407] Connection header received
0619 16:50:22.920947 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:22.920950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:22.921009 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:22.921108 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:22.922283 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.922826 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.923996 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.924226 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.924347 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.927286 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:22.927289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:22.927291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:22.927314 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:22.928765 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:22.929306 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:22.929314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:22.929335 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:22.929428 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:22.929783 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:22.929789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:22.929793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:22.930048 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:22.930150 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:22.930650 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 16:50:22.930793 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:50:22.931846 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:22.919381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:37053 (local address 127.3.233.130:57905)
0619 16:50:22.919851 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:22.919894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:22.920721 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:22.921294 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:22.921302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:22.921653 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:22.922131 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.922150 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.922966 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.922989 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.923873 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:22.923882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:22.924409 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:22.924413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:22.924578 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:22.925412 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:22.925429 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:22.927111 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:22.928900 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:22.928910 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:22.928927 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:22.929193 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:22.929536 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:22.929538 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:22.929539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:22.929677 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:22.930149 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:22.930154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:22.930395 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 16:50:22.930548 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 16:50:22.930881 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260619 16:50:22.933218 23529 heartbeater.cc:347] Connected to a master server at 127.3.233.190:37053
I20260619 16:50:22.933512 23529 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:22.934120 23529 heartbeater.cc:511] Master 127.3.233.190:37053 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.233.131@KRBTEST.COM; defaulting to no policy
I20260619 16:50:22.935361 23204 ts_manager.cc:194] Registered new tserver with Master: f366098573dc4b2fa4a6de45f490beaa (127.3.233.130:46209)
I20260619 16:50:22.936036 23204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.130@KRBTEST.COM'} at 127.3.233.130:57905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
Entry for principal kudu/127.3.233.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab.
I20260619 16:50:22.954072 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37053
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:45159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:23.072158 23534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:23.072441 23534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:23.072485 23534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:23.076514 23534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:23.076579 23534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:23.076642 23534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:50:23.082273 23534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:45159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.131.keytab
--principal=kudu/127.3.233.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:37053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:23.083357 23534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:23.084326 23534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:23.093953 23539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:23.094023 23534 server_base.cc:1034] running on GCE node
W20260619 16:50:23.093940 23542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:23.093941 23540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:23.094645 23534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:23.095407 23534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:23.096566 23534 hybrid_clock.cc:648] HybridClock initialized: now 1781887823096527 us; error 55 us; skew 500 ppm
Jun 19 16:50:23 dist-test-slave-cgwx krb5kdc[23159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:23.099490 23534 init.cc:377] Logged in from keytab as kudu/127.3.233.131@KRBTEST.COM (short username kudu)
I20260619 16:50:23.100383 23534 webserver.cc:460] Webserver started at http://127.3.233.131:35971/ using document root <none> and password file <none>
I20260619 16:50:23.100922 23534 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:23.100983 23534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:23.101150 23534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:23.103297 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "17bcb3d9b2434325834957bd3a0a15ae"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "c8b20d3ee10080513433e882900562f7"
server_key_iv: "d4370cb8d023c365c75a0914a9191244"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.103816 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "17bcb3d9b2434325834957bd3a0a15ae"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "c8b20d3ee10080513433e882900562f7"
server_key_iv: "d4370cb8d023c365c75a0914a9191244"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.107708 23534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:50:23.110430 23549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.111967 23534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:23.112097 23534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "17bcb3d9b2434325834957bd3a0a15ae"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "c8b20d3ee10080513433e882900562f7"
server_key_iv: "d4370cb8d023c365c75a0914a9191244"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.112201 23534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:23.126509 23534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:23.129704 23534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:23.129918 23534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:23.130556 23534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:23.131435 23534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:23.131481 23534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.131549 23534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:23.131589 23534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.140715 23534 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:42027
I20260619 16:50:23.140728 23662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:42027 every 8 connection(s)
I20260619 16:50:23.141634 23534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:23.143253 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 23534
I20260619 16:50:23.143422 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:23.143752 4006 external_mini_cluster.cc:1428] Setting key e2982714cb2aaa7b1e19c2a8ba2f48dd
Jun 19 16:50:23 dist-test-slave-cgwx krb5kdc[23159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.131@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:23.155979 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.144052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:54947 (local address 127.3.233.190:37053)
0619 16:50:23.144203 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.144207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.144911 (+ 704us) server_negotiation.cc:407] Connection header received
0619 16:50:23.146088 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.146092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.146173 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.146252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.147447 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.148064 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.149349 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.149569 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.149663 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.152380 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:23.152383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:23.152385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:23.152410 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:23.153769 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:23.154290 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:23.154292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:23.154293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:23.154364 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:23.154669 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:23.154671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:23.154672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:23.154857 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:23.155018 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.155726 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.155852 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:50:23.157203 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.144222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:37053 (local address 127.3.233.131:54947)
0619 16:50:23.144746 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.144802 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.145813 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.146401 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.146408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.146754 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:23.147313 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.147327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.148277 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.148279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.149198 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.149207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.149736 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.149739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.149905 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.150596 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:23.150612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:23.152159 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:23.153881 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:23.153886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:23.153904 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:23.154150 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:23.154449 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:23.154451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:23.154452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:23.154581 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:23.154996 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:23.155002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:23.155437 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 16:50:23.155604 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 16:50:23.155965 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260619 16:50:23.158533 23663 heartbeater.cc:347] Connected to a master server at 127.3.233.190:37053
I20260619 16:50:23.158797 23663 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:23.159322 23663 heartbeater.cc:511] Master 127.3.233.190:37053 requested a full tablet report, sending...
I20260619 16:50:23.160415 23203 ts_manager.cc:194] Registered new tserver with Master: 17bcb3d9b2434325834957bd3a0a15ae (127.3.233.131:42027)
I20260619 16:50:23.161021 23203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.233.131@KRBTEST.COM'} at 127.3.233.131:54947
I20260619 16:50:23.170454 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:23.180580 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.172677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.3.233.190:37053)
0619 16:50:23.172803 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.172807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.172968 (+ 161us) server_negotiation.cc:407] Connection header received
0619 16:50:23.173069 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.173071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.173111 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.173179 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.173887 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.174390 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.175322 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.175548 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.175632 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.177076 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:23.177080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:23.177082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:23.177165 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:23.178941 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:23.179381 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:23.179390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:23.179392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:23.179471 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:23.179757 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:23.179760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:23.179761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:23.179994 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:23.180104 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.180352 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.180462 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:50:23.183785 23203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:23.186739 23203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:23.201571 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.196392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:41257 (local address 127.0.0.1:42834)
0619 16:50:23.196874 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.196888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.196981 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.197649 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.197651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.197667 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:23.197754 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.197759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.198711 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.198716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.200069 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.200080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.201029 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.201033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.201143 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.201178 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:50:23.201295 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:50:23.201419 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":8,"thread_start_us":81,"threads_started":1}
I20260619 16:50:23.201581 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.196815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.3.233.129:41257)
0619 16:50:23.197187 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.197200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.197217 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:50:23.197310 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.197316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.197522 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.197672 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:23.197866 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.198586 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.200204 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.200898 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.201030 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.201199 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.201326 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.201419 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":106,"threads_started":1}
I20260619 16:50:23.201815 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.195640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:46209 (local address 127.0.0.1:47852)
0619 16:50:23.196214 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.196271 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.196396 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.197322 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.197328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.197353 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:23.197498 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.197505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.198997 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.199001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.200260 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.200267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.201392 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.201396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.201521 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.201542 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:50:23.201606 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:50:23.201677 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":75,"thread_start_us":85,"threads_started":1}
I20260619 16:50:23.201954 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.195794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.3.233.130:46209)
0619 16:50:23.196789 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.196801 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.196834 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:50:23.196915 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.196922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.197150 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.197313 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:23.197680 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.198823 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.200381 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.201245 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.201504 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.201650 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.201700 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.201787 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":144,"threads_started":1}
I20260619 16:50:23.202355 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.195864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:42027 (local address 127.0.0.1:43808)
0619 16:50:23.196701 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.196716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.196801 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.197475 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.197478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.197496 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:23.197604 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.197610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.199274 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.199278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.200389 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.200400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.201392 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.201396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.202031 (+ 635us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.202054 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:50:23.202153 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:50:23.202220 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":60,"thread_start_us":120,"threads_started":1}
I20260619 16:50:23.202575 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.196436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.3.233.131:42027)
0619 16:50:23.197042 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.197057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.197077 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:50:23.197146 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.197151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.197316 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.197769 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:23.197919 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.199133 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.200515 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.201260 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.202175 (+ 915us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.202313 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.202362 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.202446 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":105,"threads_started":1}
I20260619 16:50:23.205238 23329 tablet_service.cc:1467] Processing CreateTablet for tablet 59818288bd554096a13b2553f0cf4397 (DEFAULT_TABLE table=test-table [id=ec3e513552fb482bb88f52001354f71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:23.205250 23597 tablet_service.cc:1467] Processing CreateTablet for tablet 59818288bd554096a13b2553f0cf4397 (DEFAULT_TABLE table=test-table [id=ec3e513552fb482bb88f52001354f71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:23.205250 23463 tablet_service.cc:1467] Processing CreateTablet for tablet 59818288bd554096a13b2553f0cf4397 (DEFAULT_TABLE table=test-table [id=ec3e513552fb482bb88f52001354f71e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:23.206321 23463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59818288bd554096a13b2553f0cf4397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:23.206327 23597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59818288bd554096a13b2553f0cf4397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:23.206321 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59818288bd554096a13b2553f0cf4397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:23.213312 23680 tablet_bootstrap.cc:492] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df: Bootstrap starting.
I20260619 16:50:23.215003 23681 tablet_bootstrap.cc:492] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Bootstrap starting.
I20260619 16:50:23.215332 23682 tablet_bootstrap.cc:492] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Bootstrap starting.
I20260619 16:50:23.216367 23680 tablet_bootstrap.cc:654] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:23.217156 23681 tablet_bootstrap.cc:654] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:23.217419 23682 tablet_bootstrap.cc:654] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:23.218194 23680 log.cc:824] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:23.218304 23681 log.cc:824] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:23.218487 23682 log.cc:824] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:23.220136 23680 tablet_bootstrap.cc:492] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df: No bootstrap required, opened a new log
I20260619 16:50:23.220263 23682 tablet_bootstrap.cc:492] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: No bootstrap required, opened a new log
I20260619 16:50:23.220254 23681 tablet_bootstrap.cc:492] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: No bootstrap required, opened a new log
I20260619 16:50:23.220362 23680 ts_tablet_manager.cc:1397] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:50:23.220436 23681 ts_tablet_manager.cc:1397] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:50:23.220436 23682 ts_tablet_manager.cc:1397] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:50:23.223865 23680 raft_consensus.cc:348] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.224165 23680 raft_consensus.cc:374] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:23.224222 23680 raft_consensus.cc:729] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25477e4e06e04ebf998354ec285f23df, State: Initialized, Role: FOLLOWER
I20260619 16:50:23.224277 23681 raft_consensus.cc:348] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.224536 23681 raft_consensus.cc:374] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:23.224583 23681 raft_consensus.cc:729] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f366098573dc4b2fa4a6de45f490beaa, State: Initialized, Role: FOLLOWER
I20260619 16:50:23.224716 23680 consensus_queue.cc:260] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.225019 23681 consensus_queue.cc:260] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.225067 23682 raft_consensus.cc:348] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.225337 23682 raft_consensus.cc:374] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:23.225395 23682 raft_consensus.cc:729] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17bcb3d9b2434325834957bd3a0a15ae, State: Initialized, Role: FOLLOWER
I20260619 16:50:23.225422 23395 heartbeater.cc:503] Master 127.3.233.190:37053 was elected leader, sending a full tablet report...
I20260619 16:50:23.225796 23680 ts_tablet_manager.cc:1428] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:50:23.225796 23682 consensus_queue.cc:260] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.225972 23681 ts_tablet_manager.cc:1428] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:50:23.226315 23529 heartbeater.cc:503] Master 127.3.233.190:37053 was elected leader, sending a full tablet report...
I20260619 16:50:23.226579 23663 heartbeater.cc:503] Master 127.3.233.190:37053 was elected leader, sending a full tablet report...
I20260619 16:50:23.226670 23682 ts_tablet_manager.cc:1428] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:50:23.255359 23687 raft_consensus.cc:482] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:23.255560 23687 raft_consensus.cc:504] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.256922 23687 leader_election.cc:288] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25477e4e06e04ebf998354ec285f23df (127.3.233.129:41257), 17bcb3d9b2434325834957bd3a0a15ae (127.3.233.131:42027)
I20260619 16:50:23.262147 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.257139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:42027 (local address 127.3.233.130:41333)
0619 16:50:23.257350 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.257365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.257453 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.257784 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.257789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.257807 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:23.257920 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.257924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.259036 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.259039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.260554 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.260567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.261504 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.261507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.261776 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.261821 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 16:50:23.261904 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:50:23.261985 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":188}
I20260619 16:50:23.262171 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.257355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:52693 (local address 127.3.233.129:41257)
0619 16:50:23.257478 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.257482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.257698 (+ 216us) server_negotiation.cc:407] Connection header received
0619 16:50:23.257830 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.257833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.257885 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.257954 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:23.258290 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.259035 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.260715 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.261363 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.261511 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.261577 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.261967 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.262048 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:50:23.262171 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.257248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:41333 (local address 127.3.233.131:42027)
0619 16:50:23.257388 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.257392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.257408 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:23.257594 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.257597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.257640 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.257747 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:23.258066 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.258874 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.260699 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.261345 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.261508 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.261577 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.261967 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.262048 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:50:23.262154 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.257169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:41257 (local address 127.3.233.130:52693)
0619 16:50:23.257610 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.257621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.257699 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.257995 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.258000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.258019 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:23.258121 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.258125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.259128 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.259131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.260562 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.260568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.261504 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.261507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.261779 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.261827 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:50:23.261921 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:50:23.261985 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":204,"thread_start_us":126,"threads_started":1}
I20260619 16:50:23.262796 23349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59818288bd554096a13b2553f0cf4397" candidate_uuid: "f366098573dc4b2fa4a6de45f490beaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25477e4e06e04ebf998354ec285f23df" is_pre_election: true
I20260619 16:50:23.262888 23617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59818288bd554096a13b2553f0cf4397" candidate_uuid: "f366098573dc4b2fa4a6de45f490beaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bcb3d9b2434325834957bd3a0a15ae" is_pre_election: true
I20260619 16:50:23.263162 23349 raft_consensus.cc:2454] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f366098573dc4b2fa4a6de45f490beaa in term 0.
I20260619 16:50:23.263252 23617 raft_consensus.cc:2454] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f366098573dc4b2fa4a6de45f490beaa in term 0.
I20260619 16:50:23.263720 23416 leader_election.cc:302] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25477e4e06e04ebf998354ec285f23df, f366098573dc4b2fa4a6de45f490beaa; no voters:
I20260619 16:50:23.264020 23687 raft_consensus.cc:2781] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:23.264096 23687 raft_consensus.cc:482] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:23.264125 23687 raft_consensus.cc:3037] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:23.265137 23687 raft_consensus.cc:504] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.265506 23687 leader_election.cc:288] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [CANDIDATE]: Term 1 election: Requested vote from peers 25477e4e06e04ebf998354ec285f23df (127.3.233.129:41257), 17bcb3d9b2434325834957bd3a0a15ae (127.3.233.131:42027)
I20260619 16:50:23.265872 23617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59818288bd554096a13b2553f0cf4397" candidate_uuid: "f366098573dc4b2fa4a6de45f490beaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17bcb3d9b2434325834957bd3a0a15ae"
I20260619 16:50:23.265885 23349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59818288bd554096a13b2553f0cf4397" candidate_uuid: "f366098573dc4b2fa4a6de45f490beaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25477e4e06e04ebf998354ec285f23df"
I20260619 16:50:23.265981 23617 raft_consensus.cc:3037] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:23.266011 23349 raft_consensus.cc:3037] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:23.267145 23617 raft_consensus.cc:2454] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f366098573dc4b2fa4a6de45f490beaa in term 1.
I20260619 16:50:23.267144 23349 raft_consensus.cc:2454] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f366098573dc4b2fa4a6de45f490beaa in term 1.
I20260619 16:50:23.267498 23419 leader_election.cc:302] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17bcb3d9b2434325834957bd3a0a15ae, f366098573dc4b2fa4a6de45f490beaa; no voters:
I20260619 16:50:23.267654 23687 raft_consensus.cc:2781] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:23.267869 23687 raft_consensus.cc:686] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 1 LEADER]: Becoming Leader. State: Replica: f366098573dc4b2fa4a6de45f490beaa, State: Running, Role: LEADER
I20260619 16:50:23.268242 23687 consensus_queue.cc:237] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } }
I20260619 16:50:23.271437 23203 catalog_manager.cc:5495] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa reported cstate change: term changed from 0 to 1, leader changed from <none> to f366098573dc4b2fa4a6de45f490beaa (127.3.233.130). New cstate: current_term: 1 leader_uuid: "f366098573dc4b2fa4a6de45f490beaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f366098573dc4b2fa4a6de45f490beaa" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 46209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 } health_report { overall_health: UNKNOWN } } }
I20260619 16:50:23.301580 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.297109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.3.233.130:46209)
0619 16:50:23.297260 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.297266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.297793 (+ 527us) server_negotiation.cc:407] Connection header received
0619 16:50:23.297927 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.297929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.297992 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.298089 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:23.298390 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.298903 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.299883 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.300106 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.300190 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.300548 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:23.301044 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:23.301142 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.301275 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.301328 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:50:23.308259 23617 raft_consensus.cc:1261] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 1 FOLLOWER]: Refusing update from remote peer f366098573dc4b2fa4a6de45f490beaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:23.308367 23349 raft_consensus.cc:1261] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df [term 1 FOLLOWER]: Refusing update from remote peer f366098573dc4b2fa4a6de45f490beaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:23.309068 23687 consensus_queue.cc:1035] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "17bcb3d9b2434325834957bd3a0a15ae" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:23.309419 23690 consensus_queue.cc:1035] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "25477e4e06e04ebf998354ec285f23df" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:23.317492 23693 mvcc.cc:205] Tried to move back new op lower bound from 7298612524262641664 to 7298612524109185024. Current Snapshot: MvccSnapshot[applied={T|T < 7298612524262641664}]
I20260619 16:50:23.318876 23694 mvcc.cc:205] Tried to move back new op lower bound from 7298612524262641664 to 7298612524109185024. Current Snapshot: MvccSnapshot[applied={T|T < 7298612524262641664}]
I20260619 16:50:23.326753 23202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:23.326916 23202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:23.329319 23202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5244f19f669c44a1b6ed2e325b049c0e: Sending DeleteTablet for 3 replicas of tablet 59818288bd554096a13b2553f0cf4397
I20260619 16:50:23.330168 23329 tablet_service.cc:1514] Processing DeleteTablet for tablet 59818288bd554096a13b2553f0cf4397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:23 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:42834
I20260619 16:50:23.330220 23597 tablet_service.cc:1514] Processing DeleteTablet for tablet 59818288bd554096a13b2553f0cf4397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:23 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:43808
I20260619 16:50:23.330739 23702 tablet_replica.cc:330] T 59818288bd554096a13b2553f0cf4397 P 25477e4e06e04ebf998354ec285f23df: stopping tablet replica
I20260619 16:50:23.331067 23463 tablet_service.cc:1514] Processing DeleteTablet for tablet 59818288bd554096a13b2553f0cf4397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:23 UTC) from {username='kudu', principal='kudu/127.3.233.190@KRBTEST.COM'} at 127.0.0.1:47852
I20260619 16:50:23.331187 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 23266
I20260619 16:50:23.331851 23703 tablet_replica.cc:330] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: stopping tablet replica
I20260619 16:50:23.332130 23703 raft_consensus.cc:2229] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:23.332475 23703 raft_consensus.cc:2258] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:23.332933 23704 tablet_replica.cc:330] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: stopping tablet replica
I20260619 16:50:23.333227 23704 raft_consensus.cc:2229] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:23.333640 23704 raft_consensus.cc:2258] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:23.334226 23703 ts_tablet_manager.cc:1905] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:23.335479 23704 ts_tablet_manager.cc:1905] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:23.337236 23703 ts_tablet_manager.cc:1918] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:23.337409 23703 log.cc:1196] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/59818288bd554096a13b2553f0cf4397
I20260619 16:50:23.337831 23703 ts_tablet_manager.cc:1939] T 59818288bd554096a13b2553f0cf4397 P 17bcb3d9b2434325834957bd3a0a15ae: Deleting consensus metadata
I20260619 16:50:23.338368 23704 ts_tablet_manager.cc:1918] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:23.338475 23704 log.cc:1196] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887569513258-4006-0/minicluster-data/ts-1/wal/wals/59818288bd554096a13b2553f0cf4397
I20260619 16:50:23.338793 23704 ts_tablet_manager.cc:1939] T 59818288bd554096a13b2553f0cf4397 P f366098573dc4b2fa4a6de45f490beaa: Deleting consensus metadata
I20260619 16:50:23.339504 23191 catalog_manager.cc:4841] TS 17bcb3d9b2434325834957bd3a0a15ae (127.3.233.131:42027): tablet 59818288bd554096a13b2553f0cf4397 (table test-table [id=ec3e513552fb482bb88f52001354f71e]) successfully deleted
I20260619 16:50:23.339852 23188 catalog_manager.cc:4841] TS f366098573dc4b2fa4a6de45f490beaa (127.3.233.130:46209): tablet 59818288bd554096a13b2553f0cf4397 (table test-table [id=ec3e513552fb482bb88f52001354f71e]) successfully deleted
W20260619 16:50:23.340322 23188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:41257 (error 108)
I20260619 16:50:23.341182 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 23400
I20260619 16:50:23.342986 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.341152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:41257 (local address 127.0.0.1:42844)
0619 16:50:23.341353 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.341432 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:41257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260619 16:50:23.343168 23188 catalog_manager.cc:4584] TS 25477e4e06e04ebf998354ec285f23df (127.3.233.129:41257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59818288bd554096a13b2553f0cf4397: Network error: Client connection negotiation failed: client connection to 127.3.233.129:41257: connect: Connection refused (error 111)
I20260619 16:50:23.348171 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 23534
I20260619 16:50:23.353644 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 23172
2026-06-19T16:50:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2314 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T16:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:23Z Disabled control of system clock
I20260619 16:50:23.383874 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34719
--webserver_interface=127.3.233.190
--webserver_port=0
--builtin_ntp_servers=127.3.233.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:50:23.494091 23709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:23.494381 23709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:23.494416 23709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:23.498345 23709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:23.498409 23709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:23.498425 23709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:23.498440 23709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:23.498456 23709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:23.503187 23709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:34719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:23.504240 23709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:23.505568 23709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:23.514559 23714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:23.514907 23717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:23.514941 23709 server_base.cc:1034] running on GCE node
W20260619 16:50:23.514832 23715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:23.515746 23709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:23.517199 23709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:23.518416 23709 hybrid_clock.cc:648] HybridClock initialized: now 1781887823518389 us; error 70 us; skew 500 ppm
I20260619 16:50:23.520603 23709 webserver.cc:460] Webserver started at http://127.3.233.190:38565/ using document root <none> and password file <none>
I20260619 16:50:23.521214 23709 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:23.521281 23709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:23.521472 23709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:23.523696 23709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "9c4824a27b3c4c79b3eb70529fb180ae"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "38e3ce01f38e16c27a7638e6465c10cc"
server_key_iv: "b2db659a4486725f0a522b788698c3db"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.524219 23709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "9c4824a27b3c4c79b3eb70529fb180ae"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "38e3ce01f38e16c27a7638e6465c10cc"
server_key_iv: "b2db659a4486725f0a522b788698c3db"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.527932 23709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:50:23.530319 23723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.531723 23709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:50:23.531842 23709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "9c4824a27b3c4c79b3eb70529fb180ae"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "38e3ce01f38e16c27a7638e6465c10cc"
server_key_iv: "b2db659a4486725f0a522b788698c3db"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.531934 23709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:23.551304 23709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:23.552013 23709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:23.552173 23709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:23.559459 23709 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:34719
I20260619 16:50:23.559500 23775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:34719 every 8 connection(s)
I20260619 16:50:23.560482 23709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:23.563656 23776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:23.569410 23776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae: Bootstrap starting.
I20260619 16:50:23.569756 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 23709
I20260619 16:50:23.569898 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:23.570209 4006 external_mini_cluster.cc:1428] Setting key 12c9e42bd9a43ce8505c12cc6c763ae6
I20260619 16:50:23.572325 23776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:23.573231 23776 log.cc:824] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:23.575551 23776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae: No bootstrap required, opened a new log
I20260619 16:50:23.580057 23776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34719 } }
I20260619 16:50:23.580334 23776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:23.580384 23776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c4824a27b3c4c79b3eb70529fb180ae, State: Initialized, Role: FOLLOWER
I20260619 16:50:23.580860 23776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34719 } }
I20260619 16:50:23.581038 23776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:23.581104 23776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:23.581199 23776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:23.581251 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.571852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.3.233.190:34719)
0619 16:50:23.572581 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.572603 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.572648 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:50:23.573509 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.573533 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.573936 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.574289 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.574718 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.576196 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.577203 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.577915 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.578080 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.579049 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:23.579059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:23.579079 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:23.579616 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:23.579684 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:23.579709 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:23.579809 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.579976 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.580249 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260619 16:50:23.582402 23776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34719 } }
I20260619 16:50:23.582791 23776 leader_election.cc:302] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c4824a27b3c4c79b3eb70529fb180ae; no voters:
I20260619 16:50:23.583137 23776 leader_election.cc:288] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:23.583251 23781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:23.583467 23781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [term 1 LEADER]: Becoming Leader. State: Replica: 9c4824a27b3c4c79b3eb70529fb180ae, State: Running, Role: LEADER
I20260619 16:50:23.583801 23781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34719 } }
I20260619 16:50:23.584410 23776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:23.584942 23781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c4824a27b3c4c79b3eb70529fb180ae. Latest consensus state: current_term: 1 leader_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34719 } } }
I20260619 16:50:23.585052 23781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:23.585418 23786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:23.585762 23782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4824a27b3c4c79b3eb70529fb180ae" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 34719 } } }
I20260619 16:50:23.585932 23782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:23.588385 23786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:23.593621 23786 catalog_manager.cc:1347] Generated new cluster ID: 56c29993cca1428bba960c59c3a8ddea
I20260619 16:50:23.593694 23786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:23.609351 23786 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:23.610649 23786 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:23.619313 23786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae: Generated new TSK 0
I20260619 16:50:23.620272 23786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:23.625835 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.129:0
--local_ip_for_outbound_sockets=127.3.233.129
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34719
--builtin_ntp_servers=127.3.233.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:50:23.738672 23800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:23.739022 23800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:23.739106 23800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:23.743346 23800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:23.743434 23800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:23.743530 23800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.129
I20260619 16:50:23.748497 23800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.233.129
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.129
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:23.749738 23800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:23.751312 23800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:23.761492 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:23.761459 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:23.761469 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:23.761809 23800 server_base.cc:1034] running on GCE node
I20260619 16:50:23.762252 23800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:23.763036 23800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:23.764228 23800 hybrid_clock.cc:648] HybridClock initialized: now 1781887823764189 us; error 53 us; skew 500 ppm
I20260619 16:50:23.773852 23800 webserver.cc:460] Webserver started at http://127.3.233.129:45459/ using document root <none> and password file <none>
I20260619 16:50:23.774535 23800 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:23.774641 23800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:23.774870 23800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:23.777087 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data/instance:
uuid: "ece2000264b04202a225f2acaf6d5440"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "4d5054e87bfad9326aeac6d37a615615"
server_key_iv: "a3c597b5a10261ab1031d59a0d06e3a6"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.777688 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance:
uuid: "ece2000264b04202a225f2acaf6d5440"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "4d5054e87bfad9326aeac6d37a615615"
server_key_iv: "a3c597b5a10261ab1031d59a0d06e3a6"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.781658 23800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:23.784324 23814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.785801 23800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:50:23.785946 23800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/wal
uuid: "ece2000264b04202a225f2acaf6d5440"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "4d5054e87bfad9326aeac6d37a615615"
server_key_iv: "a3c597b5a10261ab1031d59a0d06e3a6"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.786065 23800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:23.813719 23800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:23.814522 23800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:23.814745 23800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:23.815423 23800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:23.816462 23800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:23.816533 23800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.816608 23800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:23.816653 23800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.826130 23800 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.129:42183
I20260619 16:50:23.826154 23927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.129:42183 every 8 connection(s)
I20260619 16:50:23.827149 23800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:23.833010 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 23800
I20260619 16:50:23.833149 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:23.833460 4006 external_mini_cluster.cc:1428] Setting key 677a7ec251d0f31840c0ecf9504b7c3f
I20260619 16:50:23.835505 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.130:0
--local_ip_for_outbound_sockets=127.3.233.130
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34719
--builtin_ntp_servers=127.3.233.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:50:23.837754 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.829106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:53237 (local address 127.3.233.190:34719)
0619 16:50:23.829279 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:23.829283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:23.830433 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 16:50:23.832216 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:23.832256 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:23.832313 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:23.832409 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:23.833662 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.834204 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.835534 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.835856 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.835972 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.837306 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:23.837308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:23.837310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:23.837323 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:23.837342 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:23.837350 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:23.837411 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:23.837592 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:50:23.837643 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:50:23.838727 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:23.829525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34719 (local address 127.3.233.129:53237)
0619 16:50:23.830212 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:23.830274 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:23.832002 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:23.832604 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:23.832611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:23.832698 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:23.833427 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.833451 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.834335 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.834341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.835414 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:23.835423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:23.836014 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:23.836018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:23.836213 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:23.836957 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:23.836961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:23.837022 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:23.837112 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:23.837437 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:23.837447 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:23.837455 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:50:23.837593 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:50:23.837925 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260619 16:50:23.840049 23928 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34719
I20260619 16:50:23.840302 23928 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:23.840889 23928 heartbeater.cc:511] Master 127.3.233.190:34719 requested a full tablet report, sending...
I20260619 16:50:23.842551 23740 ts_manager.cc:194] Registered new tserver with Master: ece2000264b04202a225f2acaf6d5440 (127.3.233.129:42183)
I20260619 16:50:23.843786 23740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.129:53237
W20260619 16:50:23.950541 23931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:23.950867 23931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:23.950958 23931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:23.955122 23931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:23.955224 23931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:23.955323 23931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.130
I20260619 16:50:23.960429 23931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.233.130
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.130
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:23.961561 23931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:23.963006 23931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:23.972657 23939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:23.972657 23937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:23.972700 23936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:23.973115 23931 server_base.cc:1034] running on GCE node
I20260619 16:50:23.973526 23931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:23.974193 23931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:23.975390 23931 hybrid_clock.cc:648] HybridClock initialized: now 1781887823975355 us; error 49 us; skew 500 ppm
I20260619 16:50:23.977345 23931 webserver.cc:460] Webserver started at http://127.3.233.130:42953/ using document root <none> and password file <none>
I20260619 16:50:23.977883 23931 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:23.977942 23931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:23.978107 23931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:23.980225 23931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data/instance:
uuid: "3be052147f1241579a1f7a1fde3261ad"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "50ab80f8400fad5ee0010547c0e87f55"
server_key_iv: "be98f141caa2174240d5f90c3d0804ec"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.980746 23931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance:
uuid: "3be052147f1241579a1f7a1fde3261ad"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "50ab80f8400fad5ee0010547c0e87f55"
server_key_iv: "be98f141caa2174240d5f90c3d0804ec"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.984758 23931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 16:50:23.987849 23945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:23.989152 23931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:50:23.989352 23931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/wal
uuid: "3be052147f1241579a1f7a1fde3261ad"
format_stamp: "Formatted at 2026-06-19 16:50:23 on dist-test-slave-cgwx"
server_key: "50ab80f8400fad5ee0010547c0e87f55"
server_key_iv: "be98f141caa2174240d5f90c3d0804ec"
server_key_version: "encryptionkey@0"
I20260619 16:50:23.989460 23931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:24.010808 23931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:24.011540 23931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:24.011703 23931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:24.012240 23931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:24.013238 23931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:24.013294 23931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:24.013336 23931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:24.013351 23931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:24.022665 23931 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.130:38741
I20260619 16:50:24.022732 24058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.130:38741 every 8 connection(s)
I20260619 16:50:24.023658 23931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:24.032766 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 23931
I20260619 16:50:24.032934 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:24.033216 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.025627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.130:48345 (local address 127.3.233.190:34719)
0619 16:50:24.025773 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.025777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.026887 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 16:50:24.028370 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.028374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.028434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.028513 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:24.029405 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.030040 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.031257 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.031484 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.031595 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.032714 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:24.032717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:24.032720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:24.032741 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:24.032758 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:24.032765 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:24.032842 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.033042 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.033104 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:50:24.033241 4006 external_mini_cluster.cc:1428] Setting key 7a81aad26a258774ca2b2f6deac2557f
I20260619 16:50:24.034458 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.025959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34719 (local address 127.3.233.130:48345)
0619 16:50:24.026709 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.026754 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:24.028145 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:24.028682 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:24.028691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:24.028780 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:24.029265 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.029277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.030173 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.030177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.031139 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.031149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.031639 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.031643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.031804 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.032452 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:24.032454 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:24.032494 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:24.032557 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:24.032857 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:24.032867 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:24.032877 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:50:24.033390 (+ 513us) client_negotiation.cc:240] Negotiation successful
0619 16:50:24.033634 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260619 16:50:24.035523 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.233.131:0
--local_ip_for_outbound_sockets=127.3.233.131
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34719
--builtin_ntp_servers=127.3.233.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:50:24.035960 24059 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34719
I20260619 16:50:24.036212 24059 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:24.036777 24059 heartbeater.cc:511] Master 127.3.233.190:34719 requested a full tablet report, sending...
I20260619 16:50:24.037882 23740 ts_manager.cc:194] Registered new tserver with Master: 3be052147f1241579a1f7a1fde3261ad (127.3.233.130:38741)
I20260619 16:50:24.038462 23740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.130:48345
W20260619 16:50:24.149875 24062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:24.150175 24062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:24.150215 24062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:24.154966 24062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:24.155049 24062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:24.155112 24062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.233.131
I20260619 16:50:24.160831 24062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.233.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.233.131
--webserver_port=0
--tserver_master_addrs=127.3.233.190:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.233.131
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:24.162003 24062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:24.163528 24062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:24.173687 24070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:24.173738 24067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:24.173772 24068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:24.174077 24062 server_base.cc:1034] running on GCE node
I20260619 16:50:24.174530 24062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:24.175292 24062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:24.176477 24062 hybrid_clock.cc:648] HybridClock initialized: now 1781887824176455 us; error 36 us; skew 500 ppm
I20260619 16:50:24.178617 24062 webserver.cc:460] Webserver started at http://127.3.233.131:41947/ using document root <none> and password file <none>
I20260619 16:50:24.179417 24062 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:24.179492 24062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:24.179740 24062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:24.181996 24062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data/instance:
uuid: "8fda1a9f0fe944199f207e44051b829d"
format_stamp: "Formatted at 2026-06-19 16:50:24 on dist-test-slave-cgwx"
server_key: "2581cce5db1ce56c4d41b3b7eecc122d"
server_key_iv: "971a8f3e7610a229107a9860a70db4b2"
server_key_version: "encryptionkey@0"
I20260619 16:50:24.182610 24062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance:
uuid: "8fda1a9f0fe944199f207e44051b829d"
format_stamp: "Formatted at 2026-06-19 16:50:24 on dist-test-slave-cgwx"
server_key: "2581cce5db1ce56c4d41b3b7eecc122d"
server_key_iv: "971a8f3e7610a229107a9860a70db4b2"
server_key_version: "encryptionkey@0"
I20260619 16:50:24.187121 24062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 16:50:24.189924 24076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:24.191538 24062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:24.191700 24062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal
uuid: "8fda1a9f0fe944199f207e44051b829d"
format_stamp: "Formatted at 2026-06-19 16:50:24 on dist-test-slave-cgwx"
server_key: "2581cce5db1ce56c4d41b3b7eecc122d"
server_key_iv: "971a8f3e7610a229107a9860a70db4b2"
server_key_version: "encryptionkey@0"
I20260619 16:50:24.191836 24062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:24.211895 24062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:24.212671 24062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:24.212920 24062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:24.213584 24062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:24.214649 24062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:24.214728 24062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:24.214810 24062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:24.214862 24062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:24.224202 24062 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.131:42375
I20260619 16:50:24.224224 24189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.131:42375 every 8 connection(s)
I20260619 16:50:24.225263 24062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:24.232513 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24062
I20260619 16:50:24.232642 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:24.232971 4006 external_mini_cluster.cc:1428] Setting key 0fabe6cff136cf46676b999dc4e63807
I20260619 16:50:24.235620 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.227261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.131:38855 (local address 127.3.233.190:34719)
0619 16:50:24.227403 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.227406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.228505 (+ 1099us) server_negotiation.cc:407] Connection header received
0619 16:50:24.229891 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.229894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.229941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.230020 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:24.231018 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.231558 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.233016 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.233383 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.233514 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.235042 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:24.235046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:24.235049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:24.235063 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:24.235087 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:24.235098 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:24.235177 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.235433 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.235497 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:50:24.236768 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.227629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:34719 (local address 127.3.233.131:38855)
0619 16:50:24.228316 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.228359 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:24.229671 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:24.230197 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:24.230204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:24.230293 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:24.230829 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.230841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.231681 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.231684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.232864 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.232876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.233700 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.233705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.233900 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.234720 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:50:24.234722 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:24.234769 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:50:24.234839 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:24.235201 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:24.235244 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:24.235251 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:50:24.235380 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:50:24.235695 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260619 16:50:24.237987 24190 heartbeater.cc:347] Connected to a master server at 127.3.233.190:34719
I20260619 16:50:24.238256 24190 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:24.238816 24190 heartbeater.cc:511] Master 127.3.233.190:34719 requested a full tablet report, sending...
I20260619 16:50:24.239933 23740 ts_manager.cc:194] Registered new tserver with Master: 8fda1a9f0fe944199f207e44051b829d (127.3.233.131:42375)
I20260619 16:50:24.240525 23740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.233.131:38855
I20260619 16:50:24.247598 4006 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:24.253811 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.249657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.3.233.190:34719)
0619 16:50:24.249832 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.249838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.249986 (+ 148us) server_negotiation.cc:407] Connection header received
0619 16:50:24.250144 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.250147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.250196 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.250304 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:24.250587 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.251217 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.252087 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.252322 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.252433 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.253327 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:24.253331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:24.253334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:24.253349 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:24.253377 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:24.253385 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:24.253466 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.253589 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.253651 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:50:24.257220 23740 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:24.260414 23740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:24.276477 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.269760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.3.233.131:42375)
0619 16:50:24.270431 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.270443 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.270895 (+ 452us) server_negotiation.cc:407] Connection header received
0619 16:50:24.271142 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.271149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.271375 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.271503 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:24.272209 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.273408 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.275151 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.275730 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.275869 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.276099 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.276188 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.276290 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":165,"threads_started":1}
I20260619 16:50:24.276468 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.269657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:42375 (local address 127.0.0.1:37234)
0619 16:50:24.270716 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.270767 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:24.270947 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:24.271720 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:24.271728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:24.271759 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:24.272068 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.272075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.273544 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.273548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.275008 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.275018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.275866 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.275869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.275990 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.276028 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:50:24.276148 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:50:24.276287 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":119,"spinlock_wait_cycles":5504,"thread_start_us":97,"threads_started":1}
I20260619 16:50:24.276859 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.269909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.3.233.130:38741)
0619 16:50:24.270636 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.270648 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.270978 (+ 330us) server_negotiation.cc:407] Connection header received
0619 16:50:24.271142 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.271149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.271366 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.271503 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:24.272209 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.273408 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.275093 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.275895 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.275989 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.276136 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.276561 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.276695 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":93,"threads_started":1}
I20260619 16:50:24.277263 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.269657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:38741 (local address 127.0.0.1:38392)
0619 16:50:24.270766 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.270779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:24.270947 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:24.271739 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:24.271741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:24.271763 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:24.272068 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.272075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.273544 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.273548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.274968 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.274979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.276184 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.276187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.276374 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.276422 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:50:24.277013 (+ 591us) client_negotiation.cc:240] Negotiation successful
0619 16:50:24.277102 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":338,"spinlock_wait_cycles":21248,"thread_start_us":124,"threads_started":1}
I20260619 16:50:24.277659 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.270764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.3.233.129:42183)
0619 16:50:24.271161 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.271174 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.271633 (+ 459us) server_negotiation.cc:407] Connection header received
0619 16:50:24.271806 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.271811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.271978 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.272130 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:24.272379 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.273125 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.274852 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.275754 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.275868 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.276097 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.276727 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.277503 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
I20260619 16:50:24.277878 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.270489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42183 (local address 127.0.0.1:56320)
0619 16:50:24.271532 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.271544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:24.271656 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:24.272119 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:24.272122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:24.272140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:24.272245 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.272256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.273243 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.273249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.274716 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.274727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.276169 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.276173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.276370 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.276418 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:50:24.276493 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:50:24.277290 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":106,"thread_start_us":101,"threads_started":1}
I20260619 16:50:24.280095 23993 tablet_service.cc:1467] Processing CreateTablet for tablet 020216b9a4874907876e83e4b83eda29 (DEFAULT_TABLE table=test-table [id=bbb330dd3a354d20abbf4c73bd1248d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:24.280095 24124 tablet_service.cc:1467] Processing CreateTablet for tablet 020216b9a4874907876e83e4b83eda29 (DEFAULT_TABLE table=test-table [id=bbb330dd3a354d20abbf4c73bd1248d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:24.280257 23862 tablet_service.cc:1467] Processing CreateTablet for tablet 020216b9a4874907876e83e4b83eda29 (DEFAULT_TABLE table=test-table [id=bbb330dd3a354d20abbf4c73bd1248d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:24.281219 23993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020216b9a4874907876e83e4b83eda29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:24.281219 24124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020216b9a4874907876e83e4b83eda29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:24.281222 23862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020216b9a4874907876e83e4b83eda29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:24.287989 24207 tablet_bootstrap.cc:492] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440: Bootstrap starting.
I20260619 16:50:24.289367 24209 tablet_bootstrap.cc:492] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Bootstrap starting.
I20260619 16:50:24.290396 24207 tablet_bootstrap.cc:654] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:24.291007 24208 tablet_bootstrap.cc:492] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad: Bootstrap starting.
I20260619 16:50:24.291344 24209 tablet_bootstrap.cc:654] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:24.291352 24207 log.cc:824] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:24.292152 24209 log.cc:824] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:24.293124 24208 tablet_bootstrap.cc:654] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:24.293885 24208 log.cc:824] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:24.294059 24207 tablet_bootstrap.cc:492] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440: No bootstrap required, opened a new log
I20260619 16:50:24.294263 24207 ts_tablet_manager.cc:1397] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:50:24.294632 24209 tablet_bootstrap.cc:492] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: No bootstrap required, opened a new log
I20260619 16:50:24.294817 24209 ts_tablet_manager.cc:1397] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:50:24.295819 24208 tablet_bootstrap.cc:492] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad: No bootstrap required, opened a new log
I20260619 16:50:24.295990 24208 ts_tablet_manager.cc:1397] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:50:24.298386 24209 raft_consensus.cc:348] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.298679 24209 raft_consensus.cc:374] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:24.298740 24209 raft_consensus.cc:729] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fda1a9f0fe944199f207e44051b829d, State: Initialized, Role: FOLLOWER
I20260619 16:50:24.298825 24207 raft_consensus.cc:348] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.299122 24207 raft_consensus.cc:374] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:24.299183 24207 raft_consensus.cc:729] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece2000264b04202a225f2acaf6d5440, State: Initialized, Role: FOLLOWER
I20260619 16:50:24.299719 24209 consensus_queue.cc:260] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.299736 24207 consensus_queue.cc:260] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.300225 24208 raft_consensus.cc:348] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.300482 24208 raft_consensus.cc:374] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:24.300478 23928 heartbeater.cc:503] Master 127.3.233.190:34719 was elected leader, sending a full tablet report...
I20260619 16:50:24.300536 24208 raft_consensus.cc:729] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be052147f1241579a1f7a1fde3261ad, State: Initialized, Role: FOLLOWER
I20260619 16:50:24.300877 24209 ts_tablet_manager.cc:1428] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:50:24.300935 24208 consensus_queue.cc:260] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.301268 24190 heartbeater.cc:503] Master 127.3.233.190:34719 was elected leader, sending a full tablet report...
I20260619 16:50:24.301695 24059 heartbeater.cc:503] Master 127.3.233.190:34719 was elected leader, sending a full tablet report...
I20260619 16:50:24.301889 24208 ts_tablet_manager.cc:1428] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 16:50:24.302631 24207 ts_tablet_manager.cc:1428] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
W20260619 16:50:24.328464 23929 tablet.cc:2369] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:24.380975 24213 raft_consensus.cc:482] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:24.381244 24213 raft_consensus.cc:504] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.382504 24213 leader_election.cc:288] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fda1a9f0fe944199f207e44051b829d (127.3.233.131:42375), 3be052147f1241579a1f7a1fde3261ad (127.3.233.130:38741)
I20260619 16:50:24.387468 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.382968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:41161 (local address 127.3.233.130:38741)
0619 16:50:24.383151 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.383156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.383384 (+ 228us) server_negotiation.cc:407] Connection header received
0619 16:50:24.383495 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.383499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.383558 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.383649 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:24.384017 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.384905 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.386407 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.386840 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.387019 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.387073 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.387247 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.387330 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 16:50:24.387470 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.383085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.233.129:47481 (local address 127.3.233.131:42375)
0619 16:50:24.383202 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.383207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.383511 (+ 304us) server_negotiation.cc:407] Connection header received
0619 16:50:24.383651 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.383654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.383702 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.383786 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:24.384229 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.384966 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.386425 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.386839 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.387030 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.387082 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.387272 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.387352 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:50:24.387492 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.382835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.131:42375 (local address 127.3.233.129:47481)
0619 16:50:24.383380 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.383397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:24.383524 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:24.383820 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:24.383822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:24.383837 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:24.384103 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.384108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.385115 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.385120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.386213 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.386235 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.387019 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.387023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.387126 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.387152 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:50:24.387255 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:50:24.387329 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":211,"thread_start_us":139,"threads_started":1}
I20260619 16:50:24.387494 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.382835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:38741 (local address 127.3.233.129:41161)
0619 16:50:24.383263 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.383279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:24.383402 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:24.383664 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:24.383667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:24.383682 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:24.383880 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.383891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.385109 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.385113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.386213 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:24.386228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.386997 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.387002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:24.387113 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.387143 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:50:24.387206 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 16:50:24.387311 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":75,"thread_start_us":121,"threads_started":1}
I20260619 16:50:24.388293 24013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020216b9a4874907876e83e4b83eda29" candidate_uuid: "ece2000264b04202a225f2acaf6d5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be052147f1241579a1f7a1fde3261ad" is_pre_election: true
I20260619 16:50:24.388377 24144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020216b9a4874907876e83e4b83eda29" candidate_uuid: "ece2000264b04202a225f2acaf6d5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fda1a9f0fe944199f207e44051b829d" is_pre_election: true
I20260619 16:50:24.388664 24013 raft_consensus.cc:2454] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece2000264b04202a225f2acaf6d5440 in term 0.
I20260619 16:50:24.388664 24144 raft_consensus.cc:2454] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ece2000264b04202a225f2acaf6d5440 in term 0.
I20260619 16:50:24.389266 23817 leader_election.cc:302] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3be052147f1241579a1f7a1fde3261ad, 8fda1a9f0fe944199f207e44051b829d, ece2000264b04202a225f2acaf6d5440; no voters:
I20260619 16:50:24.389551 24213 raft_consensus.cc:2781] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:24.389654 24213 raft_consensus.cc:482] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:24.389698 24213 raft_consensus.cc:3037] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:24.390998 24213 raft_consensus.cc:504] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.391409 24213 leader_election.cc:288] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [CANDIDATE]: Term 1 election: Requested vote from peers 8fda1a9f0fe944199f207e44051b829d (127.3.233.131:42375), 3be052147f1241579a1f7a1fde3261ad (127.3.233.130:38741)
I20260619 16:50:24.391718 24144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020216b9a4874907876e83e4b83eda29" candidate_uuid: "ece2000264b04202a225f2acaf6d5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fda1a9f0fe944199f207e44051b829d"
I20260619 16:50:24.391808 24013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020216b9a4874907876e83e4b83eda29" candidate_uuid: "ece2000264b04202a225f2acaf6d5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be052147f1241579a1f7a1fde3261ad"
I20260619 16:50:24.391871 24144 raft_consensus.cc:3037] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:24.391930 24013 raft_consensus.cc:3037] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:24.393232 24144 raft_consensus.cc:2454] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece2000264b04202a225f2acaf6d5440 in term 1.
I20260619 16:50:24.393234 24013 raft_consensus.cc:2454] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ece2000264b04202a225f2acaf6d5440 in term 1.
I20260619 16:50:24.393604 23818 leader_election.cc:302] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3be052147f1241579a1f7a1fde3261ad, ece2000264b04202a225f2acaf6d5440; no voters:
I20260619 16:50:24.393822 24213 raft_consensus.cc:2781] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:24.394062 24213 raft_consensus.cc:686] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [term 1 LEADER]: Becoming Leader. State: Replica: ece2000264b04202a225f2acaf6d5440, State: Running, Role: LEADER
I20260619 16:50:24.394472 24213 consensus_queue.cc:237] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } }
I20260619 16:50:24.397779 23737 catalog_manager.cc:5495] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 reported cstate change: term changed from 0 to 1, leader changed from <none> to ece2000264b04202a225f2acaf6d5440 (127.3.233.129). New cstate: current_term: 1 leader_uuid: "ece2000264b04202a225f2acaf6d5440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ece2000264b04202a225f2acaf6d5440" member_type: VOTER last_known_addr { host: "127.3.233.129" port: 42183 } health_report { overall_health: HEALTHY } } }
I20260619 16:50:24.432319 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.427617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.3.233.129:42183)
0619 16:50:24.427809 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:24.427815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:24.427870 (+ 55us) server_negotiation.cc:407] Connection header received
0619 16:50:24.428246 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:24.428249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:24.428360 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:24.428474 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:24.428770 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.429553 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.430736 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:24.431050 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:24.431159 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:24.431493 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:24.431935 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:24.432077 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:24.432119 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 16:50:24.432175 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:50:24.439661 24144 raft_consensus.cc:1261] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 1 FOLLOWER]: Refusing update from remote peer ece2000264b04202a225f2acaf6d5440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:24.439658 24013 raft_consensus.cc:1261] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 1 FOLLOWER]: Refusing update from remote peer ece2000264b04202a225f2acaf6d5440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:24.440400 24218 consensus_queue.cc:1035] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be052147f1241579a1f7a1fde3261ad" member_type: VOTER last_known_addr { host: "127.3.233.130" port: 38741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:24.440608 24213 consensus_queue.cc:1035] T 020216b9a4874907876e83e4b83eda29 P ece2000264b04202a225f2acaf6d5440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fda1a9f0fe944199f207e44051b829d" member_type: VOTER last_known_addr { host: "127.3.233.131" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:24.448698 24221 mvcc.cc:205] Tried to move back new op lower bound from 7298612528896970752 to 7298612528722649088. Current Snapshot: MvccSnapshot[applied={T|T < 7298612528896970752}]
I20260619 16:50:24.450244 24223 mvcc.cc:205] Tried to move back new op lower bound from 7298612528896970752 to 7298612528722649088. Current Snapshot: MvccSnapshot[applied={T|T < 7298612528896970752}]
I20260619 16:50:24.460060 23737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:24.460291 23737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:24.463764 23737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c4824a27b3c4c79b3eb70529fb180ae: Sending DeleteTablet for 3 replicas of tablet 020216b9a4874907876e83e4b83eda29
I20260619 16:50:24.464846 24124 tablet_service.cc:1514] Processing DeleteTablet for tablet 020216b9a4874907876e83e4b83eda29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:24 UTC) from {username='slave'} at 127.0.0.1:37234
I20260619 16:50:24.464885 23862 tablet_service.cc:1514] Processing DeleteTablet for tablet 020216b9a4874907876e83e4b83eda29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:24 UTC) from {username='slave'} at 127.0.0.1:56320
I20260619 16:50:24.465433 24230 tablet_replica.cc:330] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: stopping tablet replica
I20260619 16:50:24.465828 24230 raft_consensus.cc:2229] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:24.465852 23993 tablet_service.cc:1514] Processing DeleteTablet for tablet 020216b9a4874907876e83e4b83eda29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:24 UTC) from {username='slave'} at 127.0.0.1:38392
I20260619 16:50:24.466246 24230 raft_consensus.cc:2258] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:24.466403 24232 tablet_replica.cc:330] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad: stopping tablet replica
I20260619 16:50:24.466537 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 23800
I20260619 16:50:24.466777 24232 raft_consensus.cc:2229] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:24.467121 24232 raft_consensus.cc:2258] T 020216b9a4874907876e83e4b83eda29 P 3be052147f1241579a1f7a1fde3261ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:24.469792 24230 ts_tablet_manager.cc:1905] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:24.473631 24230 ts_tablet_manager.cc:1918] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:24.473790 24230 log.cc:1196] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Deleting WAL directory at /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887569513258-4006-0/minicluster-data/ts-2/wal/wals/020216b9a4874907876e83e4b83eda29
I20260619 16:50:24.474188 24230 ts_tablet_manager.cc:1939] T 020216b9a4874907876e83e4b83eda29 P 8fda1a9f0fe944199f207e44051b829d: Deleting consensus metadata
W20260619 16:50:24.474565 23726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.233.129:42183 (error 108)
I20260619 16:50:24.475294 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 23931
I20260619 16:50:24.476367 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.475438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.129:42183 (local address 127.0.0.1:56340)
0619 16:50:24.476146 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.476233 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.129:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595}
W20260619 16:50:24.476538 23726 catalog_manager.cc:4584] TS ece2000264b04202a225f2acaf6d5440 (127.3.233.129:42183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 020216b9a4874907876e83e4b83eda29: Network error: Client connection negotiation failed: client connection to 127.3.233.129:42183: connect: Connection refused (error 111)
I20260619 16:50:24.477833 23726 catalog_manager.cc:4841] TS 8fda1a9f0fe944199f207e44051b829d (127.3.233.131:42375): tablet 020216b9a4874907876e83e4b83eda29 (table test-table [id=bbb330dd3a354d20abbf4c73bd1248d5]) successfully deleted
I20260619 16:50:24.483585 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24062
I20260619 16:50:24.487955 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:24.483773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.130:38741 (local address 127.0.0.1:38396)
0619 16:50:24.487733 (+ 3960us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:24.487831 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.130:38741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3835}
I20260619 16:50:24.490453 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 23709
2026-06-19T16:50:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1135 ms)
[----------] 6 tests from EncryptionPolicyTest (10996 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:24 dist-test-slave-cgwx krb5kdc[24236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:24 dist-test-slave-cgwx krb5kdc[24236](info): set up 2 sockets
Jun 19 16:50:24 dist-test-slave-cgwx krb5kdc[24236](info): commencing operation
krb5kdc: starting...
W20260619 16:50:25.547222 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:25 dist-test-slave-cgwx krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:25.676064 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:25.795296 24249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:25.795603 24249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:25.795648 24249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:25.799942 24249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:25.800030 24249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:25.800051 24249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:25.800068 24249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:25.800086 24249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:25.806034 24249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:25.807297 24249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:25.808385 24249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:25.818218 24254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:25.818230 24255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:25.818276 24249 server_base.cc:1034] running on GCE node
W20260619 16:50:25.818121 24257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:25.819283 24249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:25.820652 24249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:25.821897 24249 hybrid_clock.cc:648] HybridClock initialized: now 1781887825821861 us; error 52 us; skew 500 ppm
Jun 19 16:50:25 dist-test-slave-cgwx krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:25.824949 24249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:25.826032 24249 webserver.cc:460] Webserver started at http://127.0.0.1:37235/ using document root <none> and password file <none>
I20260619 16:50:25.826653 24249 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:25.826753 24249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:25.827069 24249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:25.829277 24249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "e19fea5fe15e452abec278d30aa94350"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-cgwx"
server_key: "2e5424633a6097bf3e9acd226c908b46"
server_key_iv: "b424e37a23dc403501cd79cbb41d9305"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.829893 24249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "e19fea5fe15e452abec278d30aa94350"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-cgwx"
server_key: "2e5424633a6097bf3e9acd226c908b46"
server_key_iv: "b424e37a23dc403501cd79cbb41d9305"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.833801 24249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:50:25.836573 24264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:25.837901 24249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 16:50:25.838068 24249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "e19fea5fe15e452abec278d30aa94350"
format_stamp: "Formatted at 2026-06-19 16:50:25 on dist-test-slave-cgwx"
server_key: "2e5424633a6097bf3e9acd226c908b46"
server_key_iv: "b424e37a23dc403501cd79cbb41d9305"
server_key_version: "encryptionkey@0"
I20260619 16:50:25.838197 24249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:25.853083 24249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:25.856689 24249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:25.856932 24249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:25.864853 24249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46691
I20260619 16:50:25.864863 24316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46691 every 8 connection(s)
I20260619 16:50:25.865864 24249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:25.868697 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:25.873394 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24249
I20260619 16:50:25.873576 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:25.873870 4006 external_mini_cluster.cc:1428] Setting key 047e0e49104abd9514b0e70846baa16c
I20260619 16:50:25.874487 24317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350: Bootstrap starting.
I20260619 16:50:25.876951 24317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:25.877717 24317 log.cc:824] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:25.879730 24317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350: No bootstrap required, opened a new log
Jun 19 16:50:25 dist-test-slave-cgwx krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:50:25.884687 24317 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19fea5fe15e452abec278d30aa94350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260619 16:50:25.884972 24317 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:25.885037 24317 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19fea5fe15e452abec278d30aa94350, State: Initialized, Role: FOLLOWER
I20260619 16:50:25.885555 24317 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19fea5fe15e452abec278d30aa94350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260619 16:50:25.885721 24317 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:25.885800 24317 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:25.885931 24317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:25.887284 24317 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19fea5fe15e452abec278d30aa94350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260619 16:50:25.887665 24317 leader_election.cc:302] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19fea5fe15e452abec278d30aa94350; no voters:
I20260619 16:50:25.888047 24317 leader_election.cc:288] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:25.888167 24322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:25.888401 24322 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [term 1 LEADER]: Becoming Leader. State: Replica: e19fea5fe15e452abec278d30aa94350, State: Running, Role: LEADER
I20260619 16:50:25.888717 24322 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19fea5fe15e452abec278d30aa94350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260619 16:50:25.889500 24317 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:25.889992 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:25.875324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37479 (local address 127.0.0.1:46691)
0619 16:50:25.875752 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:25.875781 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:25.875817 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:50:25.876461 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:25.876479 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:25.876833 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:25.877198 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:25.878088 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.879148 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.880081 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.880565 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.880937 (+ 372us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:25.883972 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:25.883978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:25.883993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:25.884022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:25.886166 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:25.886779 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:25.886784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:25.886788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:25.886918 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:25.887457 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:25.887461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:25.887463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:25.887969 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:25.888226 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:25.888754 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 16:50:25.889000 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260619 16:50:25.890420 24323 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e19fea5fe15e452abec278d30aa94350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19fea5fe15e452abec278d30aa94350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } } }
I20260619 16:50:25.890559 24323 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:25.890563 24324 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19fea5fe15e452abec278d30aa94350. Latest consensus state: current_term: 1 leader_uuid: "e19fea5fe15e452abec278d30aa94350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19fea5fe15e452abec278d30aa94350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } } }
I20260619 16:50:25.890766 24324 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:25.890978 24331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:25.894484 24331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:25.894523 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:25.902187 24331 catalog_manager.cc:1347] Generated new cluster ID: 982408f9c91e426684b57226fa1d2745
I20260619 16:50:25.902262 24331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:25.907109 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:25.897064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57103 (local address 127.0.0.1:46691)
0619 16:50:25.897175 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:25.897180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:25.897380 (+ 200us) server_negotiation.cc:407] Connection header received
0619 16:50:25.897492 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:25.897495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:25.897544 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:25.897638 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:25.898516 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.899321 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.900549 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:25.900847 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:25.900988 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:25.903089 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:25.903093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:25.903096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:25.903131 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:25.904924 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:25.905506 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:25.905510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:25.905512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:25.905601 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:25.905967 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:25.905970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:25.905972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:25.906212 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:25.906362 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:25.906640 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 16:50:25.906751 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:50:25.912384 24331 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:25.913473 24331 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:25.927191 24331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19fea5fe15e452abec278d30aa94350: Generated new TSK 0
I20260619 16:50:25.928068 24331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:25.933808 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24249
W20260619 16:50:25.939317 24234 connection.cc:535] client connection to 127.0.0.1:46691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46691): Connection reset by peer (error 104)
2026-06-19T16:50:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:25 dist-test-slave-cgwx krb5kdc[24348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:25 dist-test-slave-cgwx krb5kdc[24348](info): set up 2 sockets
Jun 19 16:50:25 dist-test-slave-cgwx krb5kdc[24348](info): commencing operation
krb5kdc: starting...
W20260619 16:50:26.991580 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:27 dist-test-slave-cgwx krb5kdc[24348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:27.117865 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:27.231194 24361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:27.231475 24361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:27.231516 24361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:27.235477 24361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:27.235531 24361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:27.235550 24361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:27.235566 24361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:27.235582 24361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:27.240803 24361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:27.241853 24361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:27.242830 24361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:27.251273 24369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:27.251332 24366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:27.251363 24367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:27.251704 24361 server_base.cc:1034] running on GCE node
I20260619 16:50:27.252216 24361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:27.253471 24361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:27.254678 24361 hybrid_clock.cc:648] HybridClock initialized: now 1781887827254633 us; error 63 us; skew 500 ppm
Jun 19 16:50:27 dist-test-slave-cgwx krb5kdc[24348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:27.257613 24361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:27.258548 24361 webserver.cc:460] Webserver started at http://127.0.0.1:36033/ using document root <none> and password file <none>
I20260619 16:50:27.259182 24361 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:27.259262 24361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:27.259435 24361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:27.261432 24361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-cgwx"
server_key: "496fc1c44d38dc1d4d0e3254f63a9c6e"
server_key_iv: "2d9323bb6b7c512665ad04225035fdad"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.261929 24361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-cgwx"
server_key: "496fc1c44d38dc1d4d0e3254f63a9c6e"
server_key_iv: "2d9323bb6b7c512665ad04225035fdad"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.265666 24361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:50:27.268234 24376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:27.269533 24361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:27.269663 24361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3"
format_stamp: "Formatted at 2026-06-19 16:50:27 on dist-test-slave-cgwx"
server_key: "496fc1c44d38dc1d4d0e3254f63a9c6e"
server_key_iv: "2d9323bb6b7c512665ad04225035fdad"
server_key_version: "encryptionkey@0"
I20260619 16:50:27.269771 24361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:27.289489 24361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:27.293092 24361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:27.293308 24361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:27.301090 24361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41963
I20260619 16:50:27.301118 24428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41963 every 8 connection(s)
I20260619 16:50:27.302088 24361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:27.304579 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24361
I20260619 16:50:27.304731 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:27.305052 4006 external_mini_cluster.cc:1428] Setting key 6345ebee6712f6376724187edc10b644
I20260619 16:50:27.305284 24429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:27.312527 24429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3: Bootstrap starting.
I20260619 16:50:27.314111 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:27.306408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57939 (local address 127.0.0.1:41963)
0619 16:50:27.306834 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:27.306852 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:27.306895 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:50:27.307599 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:27.307616 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:27.307958 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:27.308232 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:27.308608 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:27.309568 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:27.310767 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:27.311326 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:27.311446 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:27.312391 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:27.312396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:27.312408 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:27.312847 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:27.312901 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:27.312924 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:27.313021 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:27.313200 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:50:27.313350 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":150,"threads_started":1}
I20260619 16:50:27.314862 24429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:27.315622 24429 log.cc:824] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:27.316980 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:27.317806 24429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3: No bootstrap required, opened a new log
I20260619 16:50:27.322389 24429 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260619 16:50:27.322690 24429 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:27.322751 24429 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ee7762c14e43d6b8abc0ff4105b6a3, State: Initialized, Role: FOLLOWER
I20260619 16:50:27.323325 24429 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260619 16:50:27.323521 24429 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:27.323622 24429 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:27.323669 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:27.319078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60789 (local address 127.0.0.1:41963)
0619 16:50:27.319198 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:27.319202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:27.319417 (+ 215us) server_negotiation.cc:407] Connection header received
0619 16:50:27.319535 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:27.319537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:27.319573 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:27.319638 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:27.319982 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:27.320501 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:27.321688 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:27.321951 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:27.322041 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:27.323001 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:27.323005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:27.323007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:27.323018 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:27.323043 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:27.323053 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:27.323123 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:27.323429 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:50:27.323475 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:50:27.323726 24429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:27.324642 24429 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260619 16:50:27.324997 24429 leader_election.cc:302] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8ee7762c14e43d6b8abc0ff4105b6a3; no voters:
I20260619 16:50:27.325346 24429 leader_election.cc:288] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:27.325507 24439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:27.325774 24439 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [term 1 LEADER]: Becoming Leader. State: Replica: d8ee7762c14e43d6b8abc0ff4105b6a3, State: Running, Role: LEADER
I20260619 16:50:27.326092 24439 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } }
I20260619 16:50:27.326647 24429 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:27.327956 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } } }
I20260619 16:50:27.327984 24441 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8ee7762c14e43d6b8abc0ff4105b6a3. Latest consensus state: current_term: 1 leader_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ee7762c14e43d6b8abc0ff4105b6a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41963 } } }
I20260619 16:50:27.328145 24441 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:27.328419 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3 [sys.catalog]: This master's current role is: LEADER
W20260619 16:50:27.331480 24454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:27.331553 24454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:27.331645 24449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:27.332454 24449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:27.338128 24449 catalog_manager.cc:1347] Generated new cluster ID: b534837dc3ca464d9bd2aad4f6cf10c9
I20260619 16:50:27.338200 24449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:27.359380 24449 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:27.360591 24449 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:27.373373 24449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8ee7762c14e43d6b8abc0ff4105b6a3: Generated new TSK 0
I20260619 16:50:27.374127 24449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:27.383387 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24361
W20260619 16:50:27.388209 24346 connection.cc:535] client connection to 127.0.0.1:41963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41963): Connection reset by peer (error 104)
2026-06-19T16:50:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:27 dist-test-slave-cgwx krb5kdc[24460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:27 dist-test-slave-cgwx krb5kdc[24460](info): set up 2 sockets
Jun 19 16:50:27 dist-test-slave-cgwx krb5kdc[24460](info): commencing operation
krb5kdc: starting...
W20260619 16:50:28.411320 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:28 dist-test-slave-cgwx krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:28.528220 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:28.638595 24473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:28.638875 24473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:28.638916 24473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:28.643091 24473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:28.643189 24473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:28.643246 24473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:28.643291 24473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:28.643357 24473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:28.648864 24473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:28.649971 24473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:28.651041 24473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:28.659636 24478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:28.659608 24479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:28.659652 24481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:28.659876 24473 server_base.cc:1034] running on GCE node
I20260619 16:50:28.660552 24473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:28.661672 24473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:28.662914 24473 hybrid_clock.cc:648] HybridClock initialized: now 1781887828662892 us; error 34 us; skew 500 ppm
Jun 19 16:50:28 dist-test-slave-cgwx krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:28.666040 24473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:28.667138 24473 webserver.cc:460] Webserver started at http://127.0.0.1:44751/ using document root <none> and password file <none>
I20260619 16:50:28.667706 24473 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:28.667796 24473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:28.668030 24473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:28.670048 24473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "bc2ce1fd9b244072be65531aeb092662"
format_stamp: "Formatted at 2026-06-19 16:50:28 on dist-test-slave-cgwx"
server_key: "cae1d0c03772f02f4d365a5c3591d882"
server_key_iv: "dc078d16668197306eb9a6b5cc414bd8"
server_key_version: "encryptionkey@0"
I20260619 16:50:28.670619 24473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "bc2ce1fd9b244072be65531aeb092662"
format_stamp: "Formatted at 2026-06-19 16:50:28 on dist-test-slave-cgwx"
server_key: "cae1d0c03772f02f4d365a5c3591d882"
server_key_iv: "dc078d16668197306eb9a6b5cc414bd8"
server_key_version: "encryptionkey@0"
I20260619 16:50:28.674413 24473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:50:28.677024 24488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:28.678223 24473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:50:28.678367 24473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "bc2ce1fd9b244072be65531aeb092662"
format_stamp: "Formatted at 2026-06-19 16:50:28 on dist-test-slave-cgwx"
server_key: "cae1d0c03772f02f4d365a5c3591d882"
server_key_iv: "dc078d16668197306eb9a6b5cc414bd8"
server_key_version: "encryptionkey@0"
I20260619 16:50:28.678514 24473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:28.683094 24473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:28.685887 24473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:28.686082 24473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:28.693013 24473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42329
I20260619 16:50:28.693037 24540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42329 every 8 connection(s)
I20260619 16:50:28.693926 24473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:28.694270 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24473
I20260619 16:50:28.694429 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:28.694713 4006 external_mini_cluster.cc:1428] Setting key e0cbfaea1d58da05671c70761fbbf2a8
I20260619 16:50:28.697026 24541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:28.699846 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:28.696110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43089 (local address 127.0.0.1:42329)
0619 16:50:28.696515 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:28.696529 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:28.696569 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:50:28.697266 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:28.697283 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:28.697565 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:28.697896 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.698684 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:28.698690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:28.698711 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:28.698721 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:28.698773 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:28.698792 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:28.698963 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:28.699036 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 16:50:28.699180 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
I20260619 16:50:28.702641 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:28.703037 24541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662: Bootstrap starting.
I20260619 16:50:28.705382 24541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:28.706326 24541 log.cc:824] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:28.706336 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:28.704754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44225 (local address 127.0.0.1:42329)
0619 16:50:28.704868 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:28.704875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:28.705039 (+ 164us) server_negotiation.cc:407] Connection header received
0619 16:50:28.705148 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:28.705150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:28.705177 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:28.705239 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:28.705928 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:28.705931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:28.705933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:28.705942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:28.705972 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:28.705982 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:28.706049 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:28.706162 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 16:50:28.706192 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:50:28.708220 24541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662: No bootstrap required, opened a new log
I20260619 16:50:28.711704 24541 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc2ce1fd9b244072be65531aeb092662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 16:50:28.711969 24541 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:28.712054 24541 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2ce1fd9b244072be65531aeb092662, State: Initialized, Role: FOLLOWER
I20260619 16:50:28.712530 24541 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc2ce1fd9b244072be65531aeb092662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 16:50:28.712682 24541 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:28.712751 24541 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:28.712848 24541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:28.713768 24541 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc2ce1fd9b244072be65531aeb092662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 16:50:28.714110 24541 leader_election.cc:302] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc2ce1fd9b244072be65531aeb092662; no voters:
I20260619 16:50:28.714442 24541 leader_election.cc:288] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:28.714542 24551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:28.714804 24551 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [term 1 LEADER]: Becoming Leader. State: Replica: bc2ce1fd9b244072be65531aeb092662, State: Running, Role: LEADER
I20260619 16:50:28.715165 24551 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc2ce1fd9b244072be65531aeb092662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260619 16:50:28.715691 24541 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:28.716814 24553 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc2ce1fd9b244072be65531aeb092662. Latest consensus state: current_term: 1 leader_uuid: "bc2ce1fd9b244072be65531aeb092662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc2ce1fd9b244072be65531aeb092662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260619 16:50:28.716966 24553 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:28.717331 24560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:28.717303 24552 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc2ce1fd9b244072be65531aeb092662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc2ce1fd9b244072be65531aeb092662" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260619 16:50:28.717442 24552 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:28.720561 24560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:28.726212 24560 catalog_manager.cc:1347] Generated new cluster ID: 30826b70de7b4aeb9039e05a37adbfd3
I20260619 16:50:28.726321 24560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:28.740540 24560 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:28.740859 24560 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:28.748148 24560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc2ce1fd9b244072be65531aeb092662: Generated new TSK 0
I20260619 16:50:28.748924 24560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:28.752081 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24473
2026-06-19T16:50:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:28 dist-test-slave-cgwx krb5kdc[24572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:28 dist-test-slave-cgwx krb5kdc[24572](info): set up 2 sockets
Jun 19 16:50:28 dist-test-slave-cgwx krb5kdc[24572](info): commencing operation
krb5kdc: starting...
W20260619 16:50:29.795184 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:29 dist-test-slave-cgwx krb5kdc[24572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:29.918454 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:30.041065 24585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:30.041363 24585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:30.041405 24585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:30.045653 24585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:30.045717 24585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:30.045737 24585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:30.045753 24585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:30.045771 24585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:30.051137 24585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:30.052224 24585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:30.053318 24585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:30.062302 24590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:30.062230 24593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:30.062230 24591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:30.062460 24585 server_base.cc:1034] running on GCE node
I20260619 16:50:30.063217 24585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:30.064309 24585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:30.065471 24585 hybrid_clock.cc:648] HybridClock initialized: now 1781887830065458 us; error 42 us; skew 500 ppm
Jun 19 16:50:30 dist-test-slave-cgwx krb5kdc[24572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:30.068423 24585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:30.069429 24585 webserver.cc:460] Webserver started at http://127.0.0.1:42941/ using document root <none> and password file <none>
I20260619 16:50:30.070067 24585 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:30.070130 24585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:30.070382 24585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:30.072578 24585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "69af3bdbeaed428ba093666c8a9119fe"
format_stamp: "Formatted at 2026-06-19 16:50:30 on dist-test-slave-cgwx"
server_key: "962cd06cef6dd90cef4b6f73f23fc202"
server_key_iv: "70e42eba2a80909a1064e839579d75d1"
server_key_version: "encryptionkey@0"
I20260619 16:50:30.073173 24585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "69af3bdbeaed428ba093666c8a9119fe"
format_stamp: "Formatted at 2026-06-19 16:50:30 on dist-test-slave-cgwx"
server_key: "962cd06cef6dd90cef4b6f73f23fc202"
server_key_iv: "70e42eba2a80909a1064e839579d75d1"
server_key_version: "encryptionkey@0"
I20260619 16:50:30.077095 24585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:30.079664 24600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:30.081012 24585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:30.081175 24585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "69af3bdbeaed428ba093666c8a9119fe"
format_stamp: "Formatted at 2026-06-19 16:50:30 on dist-test-slave-cgwx"
server_key: "962cd06cef6dd90cef4b6f73f23fc202"
server_key_iv: "70e42eba2a80909a1064e839579d75d1"
server_key_version: "encryptionkey@0"
I20260619 16:50:30.081306 24585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:30.090970 24585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:30.093986 24585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:30.094208 24585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:30.101996 24585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34373
I20260619 16:50:30.102054 24652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34373 every 8 connection(s)
I20260619 16:50:30.103094 24585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:30.105384 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24585
I20260619 16:50:30.105520 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:30.105844 4006 external_mini_cluster.cc:1428] Setting key bc06fa46c547f326c5614559d815e828
I20260619 16:50:30.106137 24653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:30.113068 24653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe: Bootstrap starting.
I20260619 16:50:30.115830 24653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe: Neither blocks nor log segments found. Creating new log.
Jun 19 16:50:30 dist-test-slave-cgwx krb5kdc[24572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:50:30.116672 24653 log.cc:824] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:30.119184 24653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe: No bootstrap required, opened a new log
I20260619 16:50:30.122082 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:30.107363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36311 (local address 127.0.0.1:34373)
0619 16:50:30.107766 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:30.107781 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:30.107817 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:50:30.108889 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:30.108908 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:30.109244 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:30.109524 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:30.110522 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:30.111689 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:30.112915 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:30.113481 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:30.113638 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:30.116509 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:30.116518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:30.116538 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:30.116574 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:30.118606 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:30.119168 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:30.119174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:30.119179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:30.119342 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:30.119650 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:30.119654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:30.119656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:30.120164 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:30.120383 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:30.120716 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 16:50:30.121029 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20260619 16:50:30.123739 24653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69af3bdbeaed428ba093666c8a9119fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34373 } }
I20260619 16:50:30.124017 24653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:30.124114 24653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69af3bdbeaed428ba093666c8a9119fe, State: Initialized, Role: FOLLOWER
I20260619 16:50:30.124600 24653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69af3bdbeaed428ba093666c8a9119fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34373 } }
I20260619 16:50:30.124774 24653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:30.124840 24653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:30.124936 24653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:30.125208 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:30.126003 24653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69af3bdbeaed428ba093666c8a9119fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34373 } }
I20260619 16:50:30.126407 24653 leader_election.cc:302] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69af3bdbeaed428ba093666c8a9119fe; no voters:
I20260619 16:50:30.126785 24653 leader_election.cc:288] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:30.126874 24662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:30.127149 24662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [term 1 LEADER]: Becoming Leader. State: Replica: 69af3bdbeaed428ba093666c8a9119fe, State: Running, Role: LEADER
I20260619 16:50:30.127498 24662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69af3bdbeaed428ba093666c8a9119fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34373 } }
I20260619 16:50:30.128175 24653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:30.129060 24662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69af3bdbeaed428ba093666c8a9119fe. Latest consensus state: current_term: 1 leader_uuid: "69af3bdbeaed428ba093666c8a9119fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69af3bdbeaed428ba093666c8a9119fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34373 } } }
I20260619 16:50:30.129194 24662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:30.129491 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69af3bdbeaed428ba093666c8a9119fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69af3bdbeaed428ba093666c8a9119fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34373 } } }
I20260619 16:50:30.129606 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe [sys.catalog]: This master's current role is: LEADER
W20260619 16:50:30.133363 24678 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:30.133448 24678 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:30.133519 24672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:30.134199 24672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:30.137347 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:30.127859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49141 (local address 127.0.0.1:34373)
0619 16:50:30.128006 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:30.128010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:30.128026 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:30.128064 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:30.128067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:30.128158 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:30.128232 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:30.129108 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:30.130038 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:30.131543 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:30.131908 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:30.132031 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:30.133677 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:30.133681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:30.133684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:30.133718 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:30.135453 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:30.135943 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:30.135948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:30.135951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:30.136070 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:30.136402 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:30.136405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:30.136407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:30.136636 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:30.136782 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:30.137065 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 16:50:30.137191 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:50:30.140626 24672 catalog_manager.cc:1347] Generated new cluster ID: 0bbe0877be214b458842e75f251560e0
I20260619 16:50:30.140692 24672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:30.160171 24672 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:30.161859 24672 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:30.172340 24672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69af3bdbeaed428ba093666c8a9119fe: Generated new TSK 0
I20260619 16:50:30.173192 24672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:30.181089 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24585
2026-06-19T16:50:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:30 dist-test-slave-cgwx krb5kdc[24684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:30 dist-test-slave-cgwx krb5kdc[24684](info): set up 2 sockets
Jun 19 16:50:30 dist-test-slave-cgwx krb5kdc[24684](info): commencing operation
krb5kdc: starting...
W20260619 16:50:31.247682 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:31 dist-test-slave-cgwx krb5kdc[24684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:31.372087 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:31.483024 24697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:31.483304 24697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:31.483346 24697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:31.487285 24697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:31.487347 24697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:31.487367 24697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:31.487385 24697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:31.487399 24697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:31.492588 24697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:31.493602 24697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:31.494599 24697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:31.503271 24705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:31.503266 24702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:31.503400 24703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:31.503726 24697 server_base.cc:1034] running on GCE node
I20260619 16:50:31.504287 24697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:31.505374 24697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:31.506552 24697 hybrid_clock.cc:648] HybridClock initialized: now 1781887831506529 us; error 40 us; skew 500 ppm
Jun 19 16:50:31 dist-test-slave-cgwx krb5kdc[24684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:31.509327 24697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:31.510259 24697 webserver.cc:460] Webserver started at http://127.0.0.1:39705/ using document root <none> and password file <none>
I20260619 16:50:31.510830 24697 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:31.510921 24697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:31.511190 24697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:31.513206 24697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-cgwx"
server_key: "6e4063a9b342cee9424bd6519fdb1472"
server_key_iv: "0e01d937d87ee995297a65ba28913ae9"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.513768 24697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-cgwx"
server_key: "6e4063a9b342cee9424bd6519fdb1472"
server_key_iv: "0e01d937d87ee995297a65ba28913ae9"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.517630 24697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:50:31.520221 24712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:31.521602 24697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:31.521762 24697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa"
format_stamp: "Formatted at 2026-06-19 16:50:31 on dist-test-slave-cgwx"
server_key: "6e4063a9b342cee9424bd6519fdb1472"
server_key_iv: "0e01d937d87ee995297a65ba28913ae9"
server_key_version: "encryptionkey@0"
I20260619 16:50:31.521888 24697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:31.571252 24697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:31.574747 24697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:31.575042 24697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:31.582214 24697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34395
I20260619 16:50:31.582228 24764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34395 every 8 connection(s)
I20260619 16:50:31.583240 24697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:31.586076 24765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:31.589085 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24697
I20260619 16:50:31.589232 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:31.589504 4006 external_mini_cluster.cc:1428] Setting key 446a49839968e4c36861fc7bb5f13e58
I20260619 16:50:31.592185 24765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa: Bootstrap starting.
I20260619 16:50:31.594635 24765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:31.595479 24765 log.cc:824] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:31.597689 24765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa: No bootstrap required, opened a new log
I20260619 16:50:31.598737 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:31.590842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37537 (local address 127.0.0.1:34395)
0619 16:50:31.591317 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:31.591333 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:31.591368 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:50:31.591974 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:31.592001 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:31.592372 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:31.592648 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:31.592994 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.593934 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.595200 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.595715 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.595859 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:31.596687 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:31.596693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:31.596708 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:31.597211 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:31.597278 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:31.597307 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:31.597413 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:31.597593 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 16:50:31.597809 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":150,"threads_started":1}
I20260619 16:50:31.601871 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:31.602066 24765 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260619 16:50:31.602285 24765 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:31.602355 24765 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d8aeb73bce4eadb6fb76ad357a98aa, State: Initialized, Role: FOLLOWER
I20260619 16:50:31.602813 24765 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260619 16:50:31.602972 24765 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:31.603051 24765 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:31.603171 24765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:31.604205 24765 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260619 16:50:31.604591 24765 leader_election.cc:302] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5d8aeb73bce4eadb6fb76ad357a98aa; no voters:
I20260619 16:50:31.604923 24765 leader_election.cc:288] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:31.605046 24775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:31.605254 24775 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [term 1 LEADER]: Becoming Leader. State: Replica: c5d8aeb73bce4eadb6fb76ad357a98aa, State: Running, Role: LEADER
I20260619 16:50:31.605633 24775 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260619 16:50:31.606202 24765 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:31.606863 24775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5d8aeb73bce4eadb6fb76ad357a98aa. Latest consensus state: current_term: 1 leader_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } } }
I20260619 16:50:31.607075 24775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:31.607493 24784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:31.607463 24776 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d8aeb73bce4eadb6fb76ad357a98aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } } }
I20260619 16:50:31.608458 24776 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:31.609742 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:31.604082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34515 (local address 127.0.0.1:34395)
0619 16:50:31.604201 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:31.604205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:31.604422 (+ 217us) server_negotiation.cc:407] Connection header received
0619 16:50:31.604531 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:31.604534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:31.604583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:31.604674 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:31.605113 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.605907 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.607736 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:31.608130 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:31.608218 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:31.609241 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:31.609245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:31.609248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:31.609269 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:31.609299 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:31.609315 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:31.609401 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:31.609547 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 16:50:31.609606 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:50:31.611585 24784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:31.617609 24784 catalog_manager.cc:1347] Generated new cluster ID: fed89249aa9b4ea7b25f2042a9a28a2b
I20260619 16:50:31.617697 24784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:31.642504 24784 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:31.643540 24784 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:31.661702 24784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5d8aeb73bce4eadb6fb76ad357a98aa: Generated new TSK 0
I20260619 16:50:31.662469 24784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:31.670528 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24697
2026-06-19T16:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:31 dist-test-slave-cgwx krb5kdc[24796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:31 dist-test-slave-cgwx krb5kdc[24796](info): set up 2 sockets
Jun 19 16:50:31 dist-test-slave-cgwx krb5kdc[24796](info): commencing operation
krb5kdc: starting...
W20260619 16:50:32.723232 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:32 dist-test-slave-cgwx krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:32.848016 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:32.960608 24809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:32.960928 24809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:32.961011 24809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:32.965225 24809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:32.965327 24809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:32.965366 24809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:32.965420 24809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:32.965456 24809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:32.971083 24809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:32.972187 24809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:32.973263 24809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:32.981832 24814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:32.981853 24817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:32.981853 24815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:32.982220 24809 server_base.cc:1034] running on GCE node
I20260619 16:50:32.982818 24809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:32.984012 24809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:32.985216 24809 hybrid_clock.cc:648] HybridClock initialized: now 1781887832985201 us; error 31 us; skew 500 ppm
Jun 19 16:50:32 dist-test-slave-cgwx krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:32.988417 24809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:32.989367 24809 webserver.cc:460] Webserver started at http://127.0.0.1:38707/ using document root <none> and password file <none>
I20260619 16:50:32.989887 24809 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:32.989945 24809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:32.990114 24809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:32.992287 24809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "63ae8e72165f4c52a728f05dc081d18d"
format_stamp: "Formatted at 2026-06-19 16:50:32 on dist-test-slave-cgwx"
server_key: "2174fb859e844437e6cb1be12eface1b"
server_key_iv: "4c7afefa3f8e8d70155b5d18dd9da9e2"
server_key_version: "encryptionkey@0"
I20260619 16:50:32.992870 24809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "63ae8e72165f4c52a728f05dc081d18d"
format_stamp: "Formatted at 2026-06-19 16:50:32 on dist-test-slave-cgwx"
server_key: "2174fb859e844437e6cb1be12eface1b"
server_key_iv: "4c7afefa3f8e8d70155b5d18dd9da9e2"
server_key_version: "encryptionkey@0"
I20260619 16:50:32.996631 24809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:50:32.999207 24824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:33.000557 24809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:33.000687 24809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "63ae8e72165f4c52a728f05dc081d18d"
format_stamp: "Formatted at 2026-06-19 16:50:32 on dist-test-slave-cgwx"
server_key: "2174fb859e844437e6cb1be12eface1b"
server_key_iv: "4c7afefa3f8e8d70155b5d18dd9da9e2"
server_key_version: "encryptionkey@0"
I20260619 16:50:33.000799 24809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:33.005522 24809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:33.008601 24809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:33.008791 24809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:33.015972 24809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41101
I20260619 16:50:33.015996 24876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41101 every 8 connection(s)
I20260619 16:50:33.016957 24809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:33.019752 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:33.024220 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24809
I20260619 16:50:33.024366 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:33.024667 4006 external_mini_cluster.cc:1428] Setting key 0b5ed1afb4ae6e1dcce131cb04d0e431
I20260619 16:50:33.025702 24877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d: Bootstrap starting.
I20260619 16:50:33.028110 24877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:33.028983 24877 log.cc:824] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:33.030200 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:33.026135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44855 (local address 127.0.0.1:41101)
0619 16:50:33.026604 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:33.026618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:33.026655 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:50:33.027364 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:33.027383 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:33.027752 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:33.028102 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:33.028917 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:33.028921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:33.028932 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:33.028943 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:33.028996 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:33.029015 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:33.029128 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:33.029229 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 16:50:33.029389 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260619 16:50:33.031037 24877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d: No bootstrap required, opened a new log
I20260619 16:50:33.032953 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:33.035147 24877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ae8e72165f4c52a728f05dc081d18d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260619 16:50:33.035431 24877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:33.035493 24877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ae8e72165f4c52a728f05dc081d18d, State: Initialized, Role: FOLLOWER
I20260619 16:50:33.035993 24877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ae8e72165f4c52a728f05dc081d18d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260619 16:50:33.036175 24877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:33.036254 24877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:33.036366 24877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:33.036931 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:33.035160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36529 (local address 127.0.0.1:41101)
0619 16:50:33.035258 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:33.035264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:33.035445 (+ 181us) server_negotiation.cc:407] Connection header received
0619 16:50:33.035627 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:33.035631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:33.035669 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:33.035755 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:33.036564 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:33.036571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:33.036576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:33.036592 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:33.036622 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:33.036634 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:33.036725 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:33.036786 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 16:50:33.036822 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:50:33.037468 24877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ae8e72165f4c52a728f05dc081d18d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260619 16:50:33.037810 24877 leader_election.cc:302] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ae8e72165f4c52a728f05dc081d18d; no voters:
I20260619 16:50:33.038111 24877 leader_election.cc:288] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:33.038204 24887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:33.038429 24887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [term 1 LEADER]: Becoming Leader. State: Replica: 63ae8e72165f4c52a728f05dc081d18d, State: Running, Role: LEADER
I20260619 16:50:33.038733 24887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ae8e72165f4c52a728f05dc081d18d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260619 16:50:33.039469 24877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:33.040642 24888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ae8e72165f4c52a728f05dc081d18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ae8e72165f4c52a728f05dc081d18d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } } }
I20260619 16:50:33.040679 24889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ae8e72165f4c52a728f05dc081d18d. Latest consensus state: current_term: 1 leader_uuid: "63ae8e72165f4c52a728f05dc081d18d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ae8e72165f4c52a728f05dc081d18d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } } }
I20260619 16:50:33.040813 24888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:33.040831 24889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:33.041801 24896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:33.044698 24896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:33.050081 24896 catalog_manager.cc:1347] Generated new cluster ID: 3299f8908a8f414db8c7cfd55a20443c
I20260619 16:50:33.050163 24896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:33.069602 24896 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:33.069974 24896 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:33.083079 24896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ae8e72165f4c52a728f05dc081d18d: Generated new TSK 0
I20260619 16:50:33.083853 24896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:33.091471 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24809
2026-06-19T16:50:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:33 dist-test-slave-cgwx krb5kdc[24908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:33 dist-test-slave-cgwx krb5kdc[24908](info): set up 2 sockets
Jun 19 16:50:33 dist-test-slave-cgwx krb5kdc[24908](info): commencing operation
krb5kdc: starting...
W20260619 16:50:34.127251 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:34 dist-test-slave-cgwx krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:34.253532 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:34.363910 24921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:34.364195 24921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:34.364239 24921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:34.368304 24921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:34.368369 24921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:34.368388 24921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:34.368405 24921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:34.368420 24921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:34.373729 24921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:34.374809 24921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:34.375835 24921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:50:34.384369 24921 server_base.cc:1034] running on GCE node
W20260619 16:50:34.384353 24926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:34.384351 24927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:34.384351 24929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:34.385362 24921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:34.386498 24921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:34.387663 24921 hybrid_clock.cc:648] HybridClock initialized: now 1781887834387634 us; error 42 us; skew 500 ppm
Jun 19 16:50:34 dist-test-slave-cgwx krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:34.398095 24921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:34.399240 24921 webserver.cc:460] Webserver started at http://127.0.0.1:46001/ using document root <none> and password file <none>
I20260619 16:50:34.399894 24921 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:34.399966 24921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:34.400154 24921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:34.402385 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "6ce834a462c14747a440e0ee1992d9f1"
format_stamp: "Formatted at 2026-06-19 16:50:34 on dist-test-slave-cgwx"
server_key: "a71e851dd9a7c7898167b787021168ab"
server_key_iv: "f699bca80fce2c96537828967ef544ad"
server_key_version: "encryptionkey@0"
I20260619 16:50:34.403028 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "6ce834a462c14747a440e0ee1992d9f1"
format_stamp: "Formatted at 2026-06-19 16:50:34 on dist-test-slave-cgwx"
server_key: "a71e851dd9a7c7898167b787021168ab"
server_key_iv: "f699bca80fce2c96537828967ef544ad"
server_key_version: "encryptionkey@0"
I20260619 16:50:34.407132 24921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:50:34.409982 24936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:34.411566 24921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:34.411726 24921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "6ce834a462c14747a440e0ee1992d9f1"
format_stamp: "Formatted at 2026-06-19 16:50:34 on dist-test-slave-cgwx"
server_key: "a71e851dd9a7c7898167b787021168ab"
server_key_iv: "f699bca80fce2c96537828967ef544ad"
server_key_version: "encryptionkey@0"
I20260619 16:50:34.411902 24921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:34.427269 24921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:34.430707 24921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:34.430949 24921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:34.438068 24921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43139
I20260619 16:50:34.438089 24988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43139 every 8 connection(s)
I20260619 16:50:34.439059 24921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:34.440490 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 24921
I20260619 16:50:34.440644 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:34.440941 4006 external_mini_cluster.cc:1428] Setting key 8d34af37f38deda3ab4d9dad283b4281
I20260619 16:50:34.442298 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:34.447827 24989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1: Bootstrap starting.
Jun 19 16:50:34 dist-test-slave-cgwx krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:50:34.450194 24989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:34.450901 24989 log.cc:824] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:34.453138 24989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1: No bootstrap required, opened a new log
I20260619 16:50:34.455858 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:34.442333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:58471 (local address 127.0.0.1:43139)
0619 16:50:34.442777 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:34.442792 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:34.442829 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:50:34.443616 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:34.443632 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:34.443979 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:34.444343 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:34.445142 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:34.446136 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:34.447122 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:34.447594 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:34.447756 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:34.450793 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:34.450798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:34.450811 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:34.450838 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:34.452835 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:34.453457 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:34.453463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:34.453467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:34.453580 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:34.453918 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:34.453922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:34.453924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:34.454316 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:34.454522 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:34.454811 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 16:50:34.455081 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":137,"threads_started":1}
I20260619 16:50:34.457319 24989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce834a462c14747a440e0ee1992d9f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260619 16:50:34.457561 24989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:34.457640 24989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce834a462c14747a440e0ee1992d9f1, State: Initialized, Role: FOLLOWER
I20260619 16:50:34.458132 24989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce834a462c14747a440e0ee1992d9f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260619 16:50:34.458297 24989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:34.458371 24989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:34.458469 24989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:34.459065 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:34.459575 24989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce834a462c14747a440e0ee1992d9f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260619 16:50:34.459908 24989 leader_election.cc:302] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ce834a462c14747a440e0ee1992d9f1; no voters:
I20260619 16:50:34.460238 24989 leader_election.cc:288] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:34.460318 24994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:34.460561 24994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [term 1 LEADER]: Becoming Leader. State: Replica: 6ce834a462c14747a440e0ee1992d9f1, State: Running, Role: LEADER
I20260619 16:50:34.460896 24994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce834a462c14747a440e0ee1992d9f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } }
I20260619 16:50:34.461561 24989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:34.462463 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ce834a462c14747a440e0ee1992d9f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce834a462c14747a440e0ee1992d9f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } } }
I20260619 16:50:34.462628 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:34.462852 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ce834a462c14747a440e0ee1992d9f1. Latest consensus state: current_term: 1 leader_uuid: "6ce834a462c14747a440e0ee1992d9f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce834a462c14747a440e0ee1992d9f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43139 } } }
I20260619 16:50:34.463013 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:34.463037 25008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:34.466573 25008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:34.471170 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:34.461217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:51139 (local address 127.0.0.1:43139)
0619 16:50:34.461330 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:34.461334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:34.461627 (+ 293us) server_negotiation.cc:407] Connection header received
0619 16:50:34.461781 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:34.461784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:34.461827 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:34.461917 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:34.462709 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:34.463508 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:34.464645 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:34.464962 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:34.465050 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:34.467587 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:34.467591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:34.467593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:34.467618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:34.469323 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:34.469826 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:34.469830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:34.469831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:34.469916 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:34.470262 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:34.470265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:34.470267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:34.470449 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:34.470600 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:34.470871 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 16:50:34.471003 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:50:34.473310 25008 catalog_manager.cc:1347] Generated new cluster ID: 18fa601267384cc183a5425d62170e8b
I20260619 16:50:34.473378 25008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:34.501071 25008 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:34.502612 25008 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:34.512367 25008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ce834a462c14747a440e0ee1992d9f1: Generated new TSK 0
I20260619 16:50:34.513417 25008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:34.521999 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 24921
W20260619 16:50:34.530347 24906 connection.cc:535] client connection to 127.0.0.1:43139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43139): Connection reset by peer (error 104)
2026-06-19T16:50:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:34 dist-test-slave-cgwx krb5kdc[25020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:34 dist-test-slave-cgwx krb5kdc[25020](info): set up 2 sockets
Jun 19 16:50:34 dist-test-slave-cgwx krb5kdc[25020](info): commencing operation
krb5kdc: starting...
W20260619 16:50:35.567862 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:35 dist-test-slave-cgwx krb5kdc[25020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:35.692230 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:35.801604 25033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:35.801910 25033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:35.801954 25033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:35.806071 25033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:35.806146 25033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:35.806166 25033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:35.806183 25033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:35.806198 25033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:35.811720 25033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:35.812770 25033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:35.813733 25033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:35.822253 25038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:35.822260 25039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:35.822221 25041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:35.822435 25033 server_base.cc:1034] running on GCE node
I20260619 16:50:35.823165 25033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:35.824396 25033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:35.825588 25033 hybrid_clock.cc:648] HybridClock initialized: now 1781887835825556 us; error 49 us; skew 500 ppm
Jun 19 16:50:35 dist-test-slave-cgwx krb5kdc[25020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:35.828474 25033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:35.829335 25033 webserver.cc:460] Webserver started at http://127.0.0.1:38815/ using document root <none> and password file <none>
I20260619 16:50:35.829929 25033 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:35.830027 25033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:35.830260 25033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:35.832363 25033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "8151ce7862b243cea0c247a9a6a333c2"
format_stamp: "Formatted at 2026-06-19 16:50:35 on dist-test-slave-cgwx"
server_key: "1baae6c6ede9fb8502ff53980c30b4c4"
server_key_iv: "8d8897238f6fc9affdbef24f7b1bc9f7"
server_key_version: "encryptionkey@0"
I20260619 16:50:35.832947 25033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "8151ce7862b243cea0c247a9a6a333c2"
format_stamp: "Formatted at 2026-06-19 16:50:35 on dist-test-slave-cgwx"
server_key: "1baae6c6ede9fb8502ff53980c30b4c4"
server_key_iv: "8d8897238f6fc9affdbef24f7b1bc9f7"
server_key_version: "encryptionkey@0"
I20260619 16:50:35.836812 25033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:35.839216 25048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:35.840421 25033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:50:35.840569 25033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "8151ce7862b243cea0c247a9a6a333c2"
format_stamp: "Formatted at 2026-06-19 16:50:35 on dist-test-slave-cgwx"
server_key: "1baae6c6ede9fb8502ff53980c30b4c4"
server_key_iv: "8d8897238f6fc9affdbef24f7b1bc9f7"
server_key_version: "encryptionkey@0"
I20260619 16:50:35.840684 25033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:35.863247 25033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:35.866466 25033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:35.866705 25033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:35.874203 25033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40413
I20260619 16:50:35.874217 25100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40413 every 8 connection(s)
I20260619 16:50:35.875187 25033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:35.877956 25101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:35.878366 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25033
I20260619 16:50:35.878517 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:35.878825 4006 external_mini_cluster.cc:1428] Setting key 3180ccecc7c3d1af28d579b2261a9eee
I20260619 16:50:35.883565 25101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2: Bootstrap starting.
I20260619 16:50:35.886243 25101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:35.887027 25101 log.cc:824] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:35.888180 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:35.880168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:38963 (local address 127.0.0.1:40413)
0619 16:50:35.880578 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:35.880597 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:35.880813 (+ 216us) server_negotiation.cc:407] Connection header received
0619 16:50:35.881577 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:35.881594 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:35.881927 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:35.882251 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:35.882642 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:35.883656 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:35.884944 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:35.885394 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:35.885537 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:35.886452 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:35.886457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:35.886470 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:35.886869 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:35.886980 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:35.887022 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:35.887151 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:35.887318 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:50:35.887471 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":127,"threads_started":1}
I20260619 16:50:35.888906 25101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2: No bootstrap required, opened a new log
I20260619 16:50:35.890906 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:35.892966 25101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8151ce7862b243cea0c247a9a6a333c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260619 16:50:35.893270 25101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:35.893338 25101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8151ce7862b243cea0c247a9a6a333c2, State: Initialized, Role: FOLLOWER
I20260619 16:50:35.893810 25101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8151ce7862b243cea0c247a9a6a333c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260619 16:50:35.893976 25101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:35.894071 25101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:35.894189 25101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:35.895236 25101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8151ce7862b243cea0c247a9a6a333c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260619 16:50:35.895579 25101 leader_election.cc:302] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8151ce7862b243cea0c247a9a6a333c2; no voters:
I20260619 16:50:35.895892 25101 leader_election.cc:288] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:35.895996 25111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:35.896196 25111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [term 1 LEADER]: Becoming Leader. State: Replica: 8151ce7862b243cea0c247a9a6a333c2, State: Running, Role: LEADER
I20260619 16:50:35.896557 25111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8151ce7862b243cea0c247a9a6a333c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } }
I20260619 16:50:35.897202 25101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:35.897819 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:35.893145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:39385 (local address 127.0.0.1:40413)
0619 16:50:35.893268 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:35.893272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:35.893516 (+ 244us) server_negotiation.cc:407] Connection header received
0619 16:50:35.893617 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:35.893620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:35.893668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:35.893730 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:35.894064 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:35.894698 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:35.895847 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:35.896225 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:35.896324 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:35.897301 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:35.897304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:35.897307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:35.897319 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:35.897346 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:35.897357 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:35.897446 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:35.897659 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 16:50:35.897691 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:50:35.898262 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8151ce7862b243cea0c247a9a6a333c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8151ce7862b243cea0c247a9a6a333c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } } }
I20260619 16:50:35.898371 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:35.898747 25120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:35.898726 25113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8151ce7862b243cea0c247a9a6a333c2. Latest consensus state: current_term: 1 leader_uuid: "8151ce7862b243cea0c247a9a6a333c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8151ce7862b243cea0c247a9a6a333c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40413 } } }
I20260619 16:50:35.898862 25113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:35.901937 25120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:35.907407 25120 catalog_manager.cc:1347] Generated new cluster ID: c67e4773e5f041bebfada4a4bf5c16d2
I20260619 16:50:35.907487 25120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:35.915747 25120 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:35.916741 25120 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:35.929082 25120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8151ce7862b243cea0c247a9a6a333c2: Generated new TSK 0
I20260619 16:50:35.929857 25120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:35.935544 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25033
W20260619 16:50:35.940250 25018 connection.cc:535] client connection to 127.0.0.1:40413 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40413): Connection reset by peer (error 104)
2026-06-19T16:50:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:35 dist-test-slave-cgwx krb5kdc[25132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:35 dist-test-slave-cgwx krb5kdc[25132](info): set up 2 sockets
Jun 19 16:50:35 dist-test-slave-cgwx krb5kdc[25132](info): commencing operation
krb5kdc: starting...
W20260619 16:50:36.979159 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:37 dist-test-slave-cgwx krb5kdc[25132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:37.102507 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:37.211942 25145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:37.212266 25145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:37.212343 25145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:37.216409 25145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:37.216506 25145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:37.216543 25145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:37.216602 25145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:37.216639 25145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:37.222015 25145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:37.223125 25145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:37.224174 25145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:37.232442 25151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:37.232456 25153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:37.232457 25150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:37.232651 25145 server_base.cc:1034] running on GCE node
I20260619 16:50:37.233343 25145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:37.234457 25145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:37.235623 25145 hybrid_clock.cc:648] HybridClock initialized: now 1781887837235594 us; error 43 us; skew 500 ppm
Jun 19 16:50:37 dist-test-slave-cgwx krb5kdc[25132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:37.238533 25145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:37.239455 25145 webserver.cc:460] Webserver started at http://127.0.0.1:44229/ using document root <none> and password file <none>
I20260619 16:50:37.240020 25145 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:37.240108 25145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:37.240330 25145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:37.242347 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "02303a1509cc47aeb396de3eaf8788f0"
format_stamp: "Formatted at 2026-06-19 16:50:37 on dist-test-slave-cgwx"
server_key: "bb47397863ec009062ef7a2c970550dc"
server_key_iv: "8b63271f1de5b4d431e82dee9db7a3a4"
server_key_version: "encryptionkey@0"
I20260619 16:50:37.242883 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "02303a1509cc47aeb396de3eaf8788f0"
format_stamp: "Formatted at 2026-06-19 16:50:37 on dist-test-slave-cgwx"
server_key: "bb47397863ec009062ef7a2c970550dc"
server_key_iv: "8b63271f1de5b4d431e82dee9db7a3a4"
server_key_version: "encryptionkey@0"
I20260619 16:50:37.246692 25145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:50:37.249174 25160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:37.250571 25145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:50:37.250716 25145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "02303a1509cc47aeb396de3eaf8788f0"
format_stamp: "Formatted at 2026-06-19 16:50:37 on dist-test-slave-cgwx"
server_key: "bb47397863ec009062ef7a2c970550dc"
server_key_iv: "8b63271f1de5b4d431e82dee9db7a3a4"
server_key_version: "encryptionkey@0"
I20260619 16:50:37.250830 25145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:37.255398 25145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:37.258664 25145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:37.258848 25145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:37.265823 25145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44949
I20260619 16:50:37.265836 25212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44949 every 8 connection(s)
I20260619 16:50:37.266762 25145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:37.268492 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25145
I20260619 16:50:37.268653 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:37.268949 4006 external_mini_cluster.cc:1428] Setting key 916d135249c62aba48c55006bd2f7af6
I20260619 16:50:37.269810 25213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:37.274643 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:37.270383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:54129 (local address 127.0.0.1:44949)
0619 16:50:37.270786 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:37.270805 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:37.270846 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:50:37.271765 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:37.271788 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:37.272142 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:37.272519 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:37.273253 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:37.273260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:37.273275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:37.273287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:37.273350 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:37.273378 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:37.273526 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:37.273616 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 16:50:37.273762 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260619 16:50:37.276717 25213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0: Bootstrap starting.
I20260619 16:50:37.277624 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:37.279114 25213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:37.279903 25213 log.cc:824] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:37.281482 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:37.279774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:41009 (local address 127.0.0.1:44949)
0619 16:50:37.279907 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:37.279911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:37.280067 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:50:37.280205 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:37.280208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:37.280238 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:37.280322 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:37.281086 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:37.281088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:37.281090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:37.281101 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:37.281123 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:37.281134 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:37.281203 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:37.281305 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 16:50:37.281337 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:50:37.282125 25213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0: No bootstrap required, opened a new log
I20260619 16:50:37.285532 25213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02303a1509cc47aeb396de3eaf8788f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260619 16:50:37.285769 25213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:37.285837 25213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02303a1509cc47aeb396de3eaf8788f0, State: Initialized, Role: FOLLOWER
I20260619 16:50:37.286242 25213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02303a1509cc47aeb396de3eaf8788f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260619 16:50:37.286386 25213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:37.286474 25213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:37.286571 25213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:37.287531 25213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02303a1509cc47aeb396de3eaf8788f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260619 16:50:37.287855 25213 leader_election.cc:302] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02303a1509cc47aeb396de3eaf8788f0; no voters:
I20260619 16:50:37.288189 25213 leader_election.cc:288] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:37.288291 25223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:37.288528 25223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [term 1 LEADER]: Becoming Leader. State: Replica: 02303a1509cc47aeb396de3eaf8788f0, State: Running, Role: LEADER
I20260619 16:50:37.288818 25223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02303a1509cc47aeb396de3eaf8788f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260619 16:50:37.289356 25213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:37.290486 25225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02303a1509cc47aeb396de3eaf8788f0. Latest consensus state: current_term: 1 leader_uuid: "02303a1509cc47aeb396de3eaf8788f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02303a1509cc47aeb396de3eaf8788f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } } }
I20260619 16:50:37.290620 25225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:37.291014 25232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:37.290988 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02303a1509cc47aeb396de3eaf8788f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02303a1509cc47aeb396de3eaf8788f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } } }
I20260619 16:50:37.291082 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:37.294119 25232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:37.299960 25232 catalog_manager.cc:1347] Generated new cluster ID: e6781d9b4c03412b827b9f712d0fe617
I20260619 16:50:37.300073 25232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:37.308708 25232 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:37.309048 25232 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:37.327451 25232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02303a1509cc47aeb396de3eaf8788f0: Generated new TSK 0
I20260619 16:50:37.328251 25232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:37.338263 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25145
2026-06-19T16:50:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:37 dist-test-slave-cgwx krb5kdc[25244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:37 dist-test-slave-cgwx krb5kdc[25244](info): set up 2 sockets
Jun 19 16:50:37 dist-test-slave-cgwx krb5kdc[25244](info): commencing operation
krb5kdc: starting...
W20260619 16:50:38.435225 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:38 dist-test-slave-cgwx krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:38.557708 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:38.668831 25257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:38.669174 25257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:38.669252 25257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:38.673434 25257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:38.673548 25257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:38.673584 25257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:38.673623 25257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:38.673663 25257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:38.679140 25257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:38.680289 25257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:38.681372 25257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:38.690390 25263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:38.690490 25257 server_base.cc:1034] running on GCE node
W20260619 16:50:38.690500 25262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:38.690397 25265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:38.691335 25257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:38.692541 25257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:38.693742 25257 hybrid_clock.cc:648] HybridClock initialized: now 1781887838693726 us; error 30 us; skew 500 ppm
Jun 19 16:50:38 dist-test-slave-cgwx krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:38.696648 25257 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:38.697587 25257 webserver.cc:460] Webserver started at http://127.0.0.1:35477/ using document root <none> and password file <none>
I20260619 16:50:38.698149 25257 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:38.698241 25257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:38.698472 25257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:38.700554 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "622682bc09994af1b1ad385f1be32a48"
format_stamp: "Formatted at 2026-06-19 16:50:38 on dist-test-slave-cgwx"
server_key: "92339412c1a3960e6a5e88f21f5bff35"
server_key_iv: "741e53eb800d66e1d5a645cfe8c2e63a"
server_key_version: "encryptionkey@0"
I20260619 16:50:38.701117 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "622682bc09994af1b1ad385f1be32a48"
format_stamp: "Formatted at 2026-06-19 16:50:38 on dist-test-slave-cgwx"
server_key: "92339412c1a3960e6a5e88f21f5bff35"
server_key_iv: "741e53eb800d66e1d5a645cfe8c2e63a"
server_key_version: "encryptionkey@0"
I20260619 16:50:38.705015 25257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:38.707697 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:38.709048 25257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:38.709190 25257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "622682bc09994af1b1ad385f1be32a48"
format_stamp: "Formatted at 2026-06-19 16:50:38 on dist-test-slave-cgwx"
server_key: "92339412c1a3960e6a5e88f21f5bff35"
server_key_iv: "741e53eb800d66e1d5a645cfe8c2e63a"
server_key_version: "encryptionkey@0"
I20260619 16:50:38.709307 25257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:38.720399 25257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:38.723815 25257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:38.724031 25257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:38.731547 25257 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42887
I20260619 16:50:38.731547 25324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42887 every 8 connection(s)
I20260619 16:50:38.732554 25257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:38.733973 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25257
I20260619 16:50:38.734148 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:38.734459 4006 external_mini_cluster.cc:1428] Setting key b819be38eb89bc244074a2d83571d51f
I20260619 16:50:38.736163 25325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:38.743854 25325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48: Bootstrap starting.
Jun 19 16:50:38 dist-test-slave-cgwx krb5kdc[25244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:50:38.747126 25325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:38.748112 25325 log.cc:824] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:38.750901 25325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48: No bootstrap required, opened a new log
I20260619 16:50:38.752241 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:38.735974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:46217 (local address 127.0.0.1:42887)
0619 16:50:38.736467 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:38.736486 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:38.736532 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:50:38.737477 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:38.737501 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:38.737997 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:38.738402 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:38.739306 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:38.740996 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:38.741990 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:38.742782 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:38.742989 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:38.746146 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:38.746153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:38.746170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:38.746205 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:38.748479 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:38.749193 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:38.749202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:38.749212 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:38.749385 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:38.749835 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:38.749839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:38.749840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:38.750373 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:38.750603 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:38.750975 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 16:50:38.751227 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":127,"threads_started":1}
I20260619 16:50:38.754432 25325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622682bc09994af1b1ad385f1be32a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42887 } }
I20260619 16:50:38.754724 25325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:38.754840 25325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622682bc09994af1b1ad385f1be32a48, State: Initialized, Role: FOLLOWER
I20260619 16:50:38.755312 25325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622682bc09994af1b1ad385f1be32a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42887 } }
I20260619 16:50:38.755486 25325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:38.755576 25325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:38.755677 25325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:38.755959 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:38.757020 25325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622682bc09994af1b1ad385f1be32a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42887 } }
I20260619 16:50:38.757443 25325 leader_election.cc:302] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 622682bc09994af1b1ad385f1be32a48; no voters:
I20260619 16:50:38.757793 25325 leader_election.cc:288] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:38.757907 25334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:38.758229 25334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [term 1 LEADER]: Becoming Leader. State: Replica: 622682bc09994af1b1ad385f1be32a48, State: Running, Role: LEADER
I20260619 16:50:38.758659 25334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622682bc09994af1b1ad385f1be32a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42887 } }
I20260619 16:50:38.759243 25325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:38.760299 25336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "622682bc09994af1b1ad385f1be32a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622682bc09994af1b1ad385f1be32a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42887 } } }
I20260619 16:50:38.760423 25336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:38.760663 25336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 622682bc09994af1b1ad385f1be32a48. Latest consensus state: current_term: 1 leader_uuid: "622682bc09994af1b1ad385f1be32a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622682bc09994af1b1ad385f1be32a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42887 } } }
I20260619 16:50:38.760738 25336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:38.761085 25344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:38.764676 25344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:38.768335 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:38.758327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:39983 (local address 127.0.0.1:42887)
0619 16:50:38.758436 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:38.758441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:38.758637 (+ 196us) server_negotiation.cc:407] Connection header received
0619 16:50:38.758828 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:38.758832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:38.758896 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:38.759002 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:38.760003 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:38.760867 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:38.762204 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:38.762541 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:38.762658 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:38.764464 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:38.764468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:38.764471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:38.764496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:38.766330 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:38.766945 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:38.766949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:38.766951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:38.767045 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:38.767426 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:38.767429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:38.767431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:38.767639 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:38.767775 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:38.768095 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 16:50:38.768191 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:50:38.770714 25344 catalog_manager.cc:1347] Generated new cluster ID: a5990c9b474d42518630aaf14b81525e
I20260619 16:50:38.770779 25344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:38.781750 25344 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:38.782713 25344 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:38.793208 25344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 622682bc09994af1b1ad385f1be32a48: Generated new TSK 0
I20260619 16:50:38.793962 25344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:38.797786 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25257
W20260619 16:50:38.803205 25242 connection.cc:535] client connection to 127.0.0.1:42887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42887): Connection reset by peer (error 104)
2026-06-19T16:50:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:38 dist-test-slave-cgwx krb5kdc[25356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:38 dist-test-slave-cgwx krb5kdc[25356](info): set up 2 sockets
Jun 19 16:50:38 dist-test-slave-cgwx krb5kdc[25356](info): commencing operation
krb5kdc: starting...
W20260619 16:50:39.911521 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:39 dist-test-slave-cgwx krb5kdc[25356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:40.038494 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:40.151685 25369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:40.151971 25369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:40.152014 25369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:40.156224 25369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:40.156287 25369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:40.156307 25369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:40.156324 25369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:40.156339 25369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:40.161743 25369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:40.162861 25369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:40.163925 25369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:50:40.173082 25369 server_base.cc:1034] running on GCE node
W20260619 16:50:40.173002 25377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:40.173034 25375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:40.173002 25374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:40.173843 25369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:40.175146 25369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:40.176321 25369 hybrid_clock.cc:648] HybridClock initialized: now 1781887840176281 us; error 56 us; skew 500 ppm
Jun 19 16:50:40 dist-test-slave-cgwx krb5kdc[25356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:40.184903 25369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:40.185866 25369 webserver.cc:460] Webserver started at http://127.0.0.1:38435/ using document root <none> and password file <none>
I20260619 16:50:40.186443 25369 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:40.186513 25369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:40.186685 25369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:40.188812 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "c46c6088e6ce460e804fbc2d1161f8d3"
format_stamp: "Formatted at 2026-06-19 16:50:40 on dist-test-slave-cgwx"
server_key: "78159012b34499f9f928625b12967b71"
server_key_iv: "201ffcddd2b7774529eeb7fc9fc9c3b5"
server_key_version: "encryptionkey@0"
I20260619 16:50:40.189328 25369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "c46c6088e6ce460e804fbc2d1161f8d3"
format_stamp: "Formatted at 2026-06-19 16:50:40 on dist-test-slave-cgwx"
server_key: "78159012b34499f9f928625b12967b71"
server_key_iv: "201ffcddd2b7774529eeb7fc9fc9c3b5"
server_key_version: "encryptionkey@0"
I20260619 16:50:40.193168 25369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:40.195890 25384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:40.197280 25369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:40.197402 25369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "c46c6088e6ce460e804fbc2d1161f8d3"
format_stamp: "Formatted at 2026-06-19 16:50:40 on dist-test-slave-cgwx"
server_key: "78159012b34499f9f928625b12967b71"
server_key_iv: "201ffcddd2b7774529eeb7fc9fc9c3b5"
server_key_version: "encryptionkey@0"
I20260619 16:50:40.197516 25369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:40.221900 25369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:40.225304 25369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:40.225557 25369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:40.232892 25369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42521
I20260619 16:50:40.232915 25436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42521 every 8 connection(s)
I20260619 16:50:40.233879 25369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:40.235823 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25369
I20260619 16:50:40.235960 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:40.236240 4006 external_mini_cluster.cc:1428] Setting key 523fba38996eb3d3d302487138bc515b
I20260619 16:50:40.236806 25437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:40.242801 25437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3: Bootstrap starting.
I20260619 16:50:40.245185 25437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:40.245193 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:40.237631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:53487 (local address 127.0.0.1:42521)
0619 16:50:40.238071 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:40.238086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:40.238120 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:50:40.238759 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:40.238775 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:40.239190 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:40.239525 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:40.239944 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:40.240937 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:40.241856 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:40.242342 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:40.242525 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:40.243510 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:40.243517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:40.243533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:40.243925 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:40.243987 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:40.244009 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:40.244153 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:40.244339 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:50:40.244484 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":144,"threads_started":1}
I20260619 16:50:40.245975 25437 log.cc:824] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:40.248167 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:40.249001 25437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3: No bootstrap required, opened a new log
I20260619 16:50:40.252947 25437 raft_consensus.cc:348] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260619 16:50:40.253259 25437 raft_consensus.cc:374] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:40.253331 25437 raft_consensus.cc:729] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46c6088e6ce460e804fbc2d1161f8d3, State: Initialized, Role: FOLLOWER
I20260619 16:50:40.253770 25437 consensus_queue.cc:260] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260619 16:50:40.253958 25437 raft_consensus.cc:388] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:40.254026 25437 raft_consensus.cc:482] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:40.254139 25437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:40.254820 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:40.250145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:35849 (local address 127.0.0.1:42521)
0619 16:50:40.250250 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:40.250254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:40.250393 (+ 139us) server_negotiation.cc:407] Connection header received
0619 16:50:40.250483 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:40.250486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:40.250529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:40.250607 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:40.250843 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:40.252088 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:40.252983 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:40.253351 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:40.253467 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:40.254423 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:40.254428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:40.254430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:40.254443 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:40.254472 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:40.254485 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:40.254578 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:40.254654 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:50:40.254690 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:50:40.255378 25437 raft_consensus.cc:504] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260619 16:50:40.255752 25437 leader_election.cc:302] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c46c6088e6ce460e804fbc2d1161f8d3; no voters:
I20260619 16:50:40.256078 25437 leader_election.cc:288] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:40.256206 25447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:40.256418 25447 raft_consensus.cc:686] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [term 1 LEADER]: Becoming Leader. State: Replica: c46c6088e6ce460e804fbc2d1161f8d3, State: Running, Role: LEADER
I20260619 16:50:40.256764 25447 consensus_queue.cc:237] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260619 16:50:40.257295 25437 sys_catalog.cc:564] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:40.258401 25448 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260619 16:50:40.258540 25448 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:40.258953 25449 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46c6088e6ce460e804fbc2d1161f8d3. Latest consensus state: current_term: 1 leader_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46c6088e6ce460e804fbc2d1161f8d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260619 16:50:40.259107 25449 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:40.259379 25456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:40.262391 25456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:40.268215 25456 catalog_manager.cc:1347] Generated new cluster ID: 9b47c5aaca8b46168addf8031cba9670
I20260619 16:50:40.268297 25456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:40.285446 25456 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:40.286465 25456 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:40.292835 25456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c46c6088e6ce460e804fbc2d1161f8d3: Generated new TSK 0
I20260619 16:50:40.293574 25456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:40.303427 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25369
W20260619 16:50:40.308749 25354 connection.cc:535] client connection to 127.0.0.1:42521 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42521): Connection reset by peer (error 104)
2026-06-19T16:50:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:40 dist-test-slave-cgwx krb5kdc[25468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:40 dist-test-slave-cgwx krb5kdc[25468](info): set up 2 sockets
Jun 19 16:50:40 dist-test-slave-cgwx krb5kdc[25468](info): commencing operation
krb5kdc: starting...
W20260619 16:50:41.347250 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:41 dist-test-slave-cgwx krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:50:41.474617 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:41.584098 25481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:41.584405 25481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:41.584482 25481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:41.589684 25481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:41.589799 25481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:41.589845 25481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:41.589880 25481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:41.589915 25481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:41.595340 25481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:41.596416 25481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:41.597450 25481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:41.606076 25487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:41.606070 25486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:41.606070 25489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:41.606194 25481 server_base.cc:1034] running on GCE node
I20260619 16:50:41.607075 25481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:41.608217 25481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:41.609385 25481 hybrid_clock.cc:648] HybridClock initialized: now 1781887841609358 us; error 40 us; skew 500 ppm
Jun 19 16:50:41 dist-test-slave-cgwx krb5kdc[25468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:41.612337 25481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:50:41.613276 25481 webserver.cc:460] Webserver started at http://127.0.0.1:37397/ using document root <none> and password file <none>
I20260619 16:50:41.613842 25481 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:41.613902 25481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:41.614113 25481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:41.616321 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "8c78939b04584859ae9b5468a7081b76"
format_stamp: "Formatted at 2026-06-19 16:50:41 on dist-test-slave-cgwx"
server_key: "05de01384a68a3c70a2a4dd5ff7adc3e"
server_key_iv: "a66a0320588f068dee7f166c61316129"
server_key_version: "encryptionkey@0"
I20260619 16:50:41.616894 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "8c78939b04584859ae9b5468a7081b76"
format_stamp: "Formatted at 2026-06-19 16:50:41 on dist-test-slave-cgwx"
server_key: "05de01384a68a3c70a2a4dd5ff7adc3e"
server_key_iv: "a66a0320588f068dee7f166c61316129"
server_key_version: "encryptionkey@0"
I20260619 16:50:41.620771 25481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:41.623289 25496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:41.624557 25481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:50:41.624796 25481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "8c78939b04584859ae9b5468a7081b76"
format_stamp: "Formatted at 2026-06-19 16:50:41 on dist-test-slave-cgwx"
server_key: "05de01384a68a3c70a2a4dd5ff7adc3e"
server_key_iv: "a66a0320588f068dee7f166c61316129"
server_key_version: "encryptionkey@0"
I20260619 16:50:41.624914 25481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:41.629561 25481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:41.632823 25481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:41.633028 25481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:41.640074 25481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41375
I20260619 16:50:41.640101 25548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41375 every 8 connection(s)
I20260619 16:50:41.641009 25481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:41.643959 25549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:41.649796 25549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76: Bootstrap starting.
I20260619 16:50:41.651175 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25481
I20260619 16:50:41.651337 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:41.651669 4006 external_mini_cluster.cc:1428] Setting key 2ff42b12604289ed200067ffd550f614
I20260619 16:50:41.652908 25549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:41.653734 25549 log.cc:824] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:41.655992 25549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76: No bootstrap required, opened a new log
I20260619 16:50:41.656977 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:41.652837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:42541 (local address 127.0.0.1:41375)
0619 16:50:41.653294 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:41.653308 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:41.653345 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:50:41.654155 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:41.654178 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:41.654557 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:41.654903 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:41.655761 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:41.655765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:41.655776 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:41.655786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:41.655837 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:41.655857 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:41.655969 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:41.656079 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 16:50:41.656250 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260619 16:50:41.659807 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:41.660128 25549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78939b04584859ae9b5468a7081b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260619 16:50:41.660362 25549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:41.660444 25549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c78939b04584859ae9b5468a7081b76, State: Initialized, Role: FOLLOWER
I20260619 16:50:41.660872 25549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78939b04584859ae9b5468a7081b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260619 16:50:41.661033 25549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:41.661109 25549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:41.661249 25549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:41.662340 25549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78939b04584859ae9b5468a7081b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260619 16:50:41.662704 25549 leader_election.cc:302] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c78939b04584859ae9b5468a7081b76; no voters:
I20260619 16:50:41.663055 25549 leader_election.cc:288] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:41.663214 25559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:41.663445 25559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [term 1 LEADER]: Becoming Leader. State: Replica: 8c78939b04584859ae9b5468a7081b76, State: Running, Role: LEADER
I20260619 16:50:41.663833 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:41.661961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.72:36659 (local address 127.0.0.1:41375)
0619 16:50:41.662076 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:41.662081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:41.662228 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:50:41.662384 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:41.662387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:41.662427 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:41.662544 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:41.663321 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:41.663324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:41.663326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:41.663337 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:41.663362 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:41.663373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:41.663520 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:41.663592 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:50:41.663634 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:50:41.663906 25559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78939b04584859ae9b5468a7081b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260619 16:50:41.664383 25549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:41.665169 25559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c78939b04584859ae9b5468a7081b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78939b04584859ae9b5468a7081b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } } }
I20260619 16:50:41.665361 25559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:41.665606 25559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c78939b04584859ae9b5468a7081b76. Latest consensus state: current_term: 1 leader_uuid: "8c78939b04584859ae9b5468a7081b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c78939b04584859ae9b5468a7081b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } } }
I20260619 16:50:41.665745 25559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:41.665748 25568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:41.668592 25568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:41.673805 25568 catalog_manager.cc:1347] Generated new cluster ID: f59240f45e074dcdab7d538b3fb112b6
I20260619 16:50:41.673882 25568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:41.688874 25568 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:41.689217 25568 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:41.702370 25568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c78939b04584859ae9b5468a7081b76: Generated new TSK 0
I20260619 16:50:41.703125 25568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:41.707437 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25481
2026-06-19T16:50:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:41 dist-test-slave-cgwx krb5kdc[25580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:41 dist-test-slave-cgwx krb5kdc[25580](info): set up 2 sockets
Jun 19 16:50:41 dist-test-slave-cgwx krb5kdc[25580](info): commencing operation
krb5kdc: starting...
W20260619 16:50:42.779369 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:42 dist-test-slave-cgwx krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:42.901858 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:40959
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:40959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:43.017648 25593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:43.017987 25593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:43.018077 25593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:43.022315 25593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:43.022446 25593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:43.022481 25593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:43.022528 25593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:43.022562 25593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:43.028126 25593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:40959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:40959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:43.029318 25593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:43.030449 25593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:43.039340 25601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:43.039300 25599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:43.039301 25598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:43.039484 25593 server_base.cc:1034] running on GCE node
I20260619 16:50:43.040098 25593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:43.041222 25593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:43.042486 25593 hybrid_clock.cc:648] HybridClock initialized: now 1781887843042469 us; error 37 us; skew 500 ppm
Jun 19 16:50:43 dist-test-slave-cgwx krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:43.045382 25593 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:43.046340 25593 webserver.cc:460] Webserver started at http://127.3.233.190:38155/ using document root <none> and password file <none>
I20260619 16:50:43.046885 25593 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:43.047013 25593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:43.047199 25593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:43.049252 25593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "f7b8a0037d1c445391865210843fb9f9"
format_stamp: "Formatted at 2026-06-19 16:50:43 on dist-test-slave-cgwx"
server_key: "bf078ab18e734e02815bb0f83660b982"
server_key_iv: "a1644753cd300e6d79022a42a9f8716f"
server_key_version: "encryptionkey@0"
I20260619 16:50:43.049758 25593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "f7b8a0037d1c445391865210843fb9f9"
format_stamp: "Formatted at 2026-06-19 16:50:43 on dist-test-slave-cgwx"
server_key: "bf078ab18e734e02815bb0f83660b982"
server_key_iv: "a1644753cd300e6d79022a42a9f8716f"
server_key_version: "encryptionkey@0"
I20260619 16:50:43.053606 25593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:50:43.056188 25608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:43.057443 25593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:43.057626 25593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "f7b8a0037d1c445391865210843fb9f9"
format_stamp: "Formatted at 2026-06-19 16:50:43 on dist-test-slave-cgwx"
server_key: "bf078ab18e734e02815bb0f83660b982"
server_key_iv: "a1644753cd300e6d79022a42a9f8716f"
server_key_version: "encryptionkey@0"
I20260619 16:50:43.057742 25593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:43.097417 25593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:43.100667 25593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:43.100881 25593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:43.108101 25593 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:40959
I20260619 16:50:43.108129 25660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:40959 every 8 connection(s)
I20260619 16:50:43.109064 25593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:43.111912 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:43.117455 25661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9: Bootstrap starting.
I20260619 16:50:43.119143 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25593
I20260619 16:50:43.119297 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:43.119596 4006 external_mini_cluster.cc:1428] Setting key 952da09ba4596428ab719ad21c4a93a8
I20260619 16:50:43.120210 25661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:43.121152 25661 log.cc:824] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:43.123253 25661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9: No bootstrap required, opened a new log
I20260619 16:50:43.126694 25661 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b8a0037d1c445391865210843fb9f9" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 40959 } }
I20260619 16:50:43.127066 25661 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:43.127151 25661 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b8a0037d1c445391865210843fb9f9, State: Initialized, Role: FOLLOWER
I20260619 16:50:43.127624 25661 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b8a0037d1c445391865210843fb9f9" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 40959 } }
I20260619 16:50:43.127775 25661 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:43.127864 25661 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:43.127962 25661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:50:43 dist-test-slave-cgwx krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:43.128955 25661 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b8a0037d1c445391865210843fb9f9" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 40959 } }
I20260619 16:50:43.129302 25661 leader_election.cc:302] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b8a0037d1c445391865210843fb9f9; no voters:
I20260619 16:50:43.129637 25661 leader_election.cc:288] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:43.129760 25666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:43.129990 25666 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [term 1 LEADER]: Becoming Leader. State: Replica: f7b8a0037d1c445391865210843fb9f9, State: Running, Role: LEADER
I20260619 16:50:43.130353 25666 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b8a0037d1c445391865210843fb9f9" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 40959 } }
I20260619 16:50:43.131119 25661 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:43.132189 25667 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b8a0037d1c445391865210843fb9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b8a0037d1c445391865210843fb9f9" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 40959 } } }
I20260619 16:50:43.132187 25668 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b8a0037d1c445391865210843fb9f9. Latest consensus state: current_term: 1 leader_uuid: "f7b8a0037d1c445391865210843fb9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b8a0037d1c445391865210843fb9f9" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 40959 } } }
I20260619 16:50:43.132344 25668 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:43.132344 25667 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:43.135360 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:43.120743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58105 (local address 127.3.233.190:40959)
0619 16:50:43.121192 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:43.121207 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:43.121241 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:50:43.122128 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:43.122150 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:43.122489 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:43.122863 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:43.123728 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:43.124686 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:43.125547 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:43.126023 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:43.126204 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:43.129120 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:43.129127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:43.129151 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:43.129179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:43.131373 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:43.132085 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:43.132093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:43.132103 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:43.132264 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:43.132695 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:43.132698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:43.132700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:43.133257 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:43.134142 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:43.134327 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 16:50:43.134577 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
W20260619 16:50:43.136128 25681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:43.136201 25681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:43.136261 25675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:43.137059 25675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:43.138202 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:43.142976 25675 catalog_manager.cc:1347] Generated new cluster ID: d03e4dd090244fecbee19a88c71762c8
I20260619 16:50:43.143041 25675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:43.149248 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:43.140291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51255 (local address 127.3.233.190:40959)
0619 16:50:43.140391 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:43.140395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:43.140635 (+ 240us) server_negotiation.cc:407] Connection header received
0619 16:50:43.140733 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:43.140736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:43.140780 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:43.140864 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:43.141712 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:43.142497 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:43.143674 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:43.144018 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:43.144115 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:43.145776 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:43.145780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:43.145783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:43.145807 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:43.147463 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:43.147994 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:43.147997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:43.147999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:43.148075 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:43.148397 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:43.148400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:43.148402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:43.148589 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:43.148713 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:43.148996 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 16:50:43.149088 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":37}
I20260619 16:50:43.194051 25675 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:43.195191 25675 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:43.202912 25675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7b8a0037d1c445391865210843fb9f9: Generated new TSK 0
I20260619 16:50:43.203806 25675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:43.216727 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25593
W20260619 16:50:43.222246 25578 connection.cc:535] client connection to 127.3.233.190:40959 recv error: Network error: failed to read from TLS socket (remote: 127.3.233.190:40959): Connection reset by peer (error 104)
2026-06-19T16:50:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1509 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:43 dist-test-slave-cgwx krb5kdc[25692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:43 dist-test-slave-cgwx krb5kdc[25692](info): set up 2 sockets
Jun 19 16:50:43 dist-test-slave-cgwx krb5kdc[25692](info): commencing operation
krb5kdc: starting...
W20260619 16:50:44.295344 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:44 dist-test-slave-cgwx krb5kdc[25692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:44.414455 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35547
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:44.527076 25705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:44.527437 25705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:44.527527 25705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:44.531637 25705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:44.531740 25705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:44.531781 25705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:44.531821 25705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:44.531852 25705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:44.537348 25705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:35547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:44.538789 25705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:44.540911 25705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:44.554221 25711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:44.554257 25710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:44.554809 25705 server_base.cc:1034] running on GCE node
W20260619 16:50:44.554224 25713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:44.555862 25705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:44.557415 25705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:44.558615 25705 hybrid_clock.cc:648] HybridClock initialized: now 1781887844558605 us; error 37 us; skew 500 ppm
Jun 19 16:50:44 dist-test-slave-cgwx krb5kdc[25692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887844, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:44.562422 25705 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:44.563759 25705 webserver.cc:460] Webserver started at http://127.3.233.190:42007/ using document root <none> and password file <none>
I20260619 16:50:44.564538 25705 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:44.564635 25705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:44.564903 25705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:44.567833 25705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "3d2cfed157394b8ebdf157fa22bb456b"
format_stamp: "Formatted at 2026-06-19 16:50:44 on dist-test-slave-cgwx"
server_key: "559cd25ca5878d4e99d9c78746f1e4fd"
server_key_iv: "e3c6c2ec027fcc8f7e28f42513aa4d45"
server_key_version: "encryptionkey@0"
I20260619 16:50:44.568423 25705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "3d2cfed157394b8ebdf157fa22bb456b"
format_stamp: "Formatted at 2026-06-19 16:50:44 on dist-test-slave-cgwx"
server_key: "559cd25ca5878d4e99d9c78746f1e4fd"
server_key_iv: "e3c6c2ec027fcc8f7e28f42513aa4d45"
server_key_version: "encryptionkey@0"
I20260619 16:50:44.573819 25705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 16:50:44.577162 25720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:44.578891 25705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:44.579121 25705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "3d2cfed157394b8ebdf157fa22bb456b"
format_stamp: "Formatted at 2026-06-19 16:50:44 on dist-test-slave-cgwx"
server_key: "559cd25ca5878d4e99d9c78746f1e4fd"
server_key_iv: "e3c6c2ec027fcc8f7e28f42513aa4d45"
server_key_version: "encryptionkey@0"
I20260619 16:50:44.579247 25705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:44.592447 25705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:44.595834 25705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:44.596067 25705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:44.604676 25705 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:35547
I20260619 16:50:44.604689 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:35547 every 8 connection(s)
I20260619 16:50:44.605672 25705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:44.608629 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:44.611194 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25705
I20260619 16:50:44.611354 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:44.611716 4006 external_mini_cluster.cc:1428] Setting key 7fb6f8768fada764b3f3edad6cdbced7
I20260619 16:50:44.614822 25773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b: Bootstrap starting.
I20260619 16:50:44.617450 25773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:44.618381 25773 log.cc:824] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:44.620713 25773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b: No bootstrap required, opened a new log
I20260619 16:50:44.622115 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:44.613611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49443 (local address 127.3.233.190:35547)
0619 16:50:44.614081 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:44.614096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:44.614132 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:50:44.614745 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:44.614761 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:44.615254 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:44.615579 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:44.615943 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:44.616947 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:44.618261 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:44.618871 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:44.619094 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:44.620069 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:44.620076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:44.620093 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:44.620597 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:44.620666 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:44.620695 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:44.620830 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:44.621055 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 16:50:44.621203 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":139,"threads_started":1}
I20260619 16:50:44.625137 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:44.625121 25773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cfed157394b8ebdf157fa22bb456b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35547 } }
I20260619 16:50:44.625379 25773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:44.625453 25773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2cfed157394b8ebdf157fa22bb456b, State: Initialized, Role: FOLLOWER
I20260619 16:50:44.625877 25773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cfed157394b8ebdf157fa22bb456b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35547 } }
I20260619 16:50:44.625995 25773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:44.626080 25773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:44.626194 25773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:44.627297 25773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cfed157394b8ebdf157fa22bb456b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35547 } }
I20260619 16:50:44.627650 25773 leader_election.cc:302] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d2cfed157394b8ebdf157fa22bb456b; no voters:
I20260619 16:50:44.627962 25773 leader_election.cc:288] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:44.628078 25783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:44.628296 25783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [term 1 LEADER]: Becoming Leader. State: Replica: 3d2cfed157394b8ebdf157fa22bb456b, State: Running, Role: LEADER
I20260619 16:50:44.628645 25783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cfed157394b8ebdf157fa22bb456b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35547 } }
I20260619 16:50:44.629249 25773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:44.629845 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d2cfed157394b8ebdf157fa22bb456b. Latest consensus state: current_term: 1 leader_uuid: "3d2cfed157394b8ebdf157fa22bb456b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cfed157394b8ebdf157fa22bb456b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35547 } } }
I20260619 16:50:44.629973 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:44.630620 25784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d2cfed157394b8ebdf157fa22bb456b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cfed157394b8ebdf157fa22bb456b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 35547 } } }
I20260619 16:50:44.630714 25784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:44.631008 25790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:44.632560 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:44.627432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42497 (local address 127.3.233.190:35547)
0619 16:50:44.627553 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:44.627557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:44.627773 (+ 216us) server_negotiation.cc:407] Connection header received
0619 16:50:44.627913 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:44.627922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:44.627989 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:44.628063 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:44.628497 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:44.629268 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:44.630426 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:44.630818 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:44.630917 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:44.632013 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:44.632017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:44.632020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:44.632033 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:44.632061 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:44.632075 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:44.632150 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:44.632373 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 16:50:44.632414 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:50:44.634253 25790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:44.640316 25790 catalog_manager.cc:1347] Generated new cluster ID: f80c141b04e149fa8e22cbf16e627f35
I20260619 16:50:44.640396 25790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:44.652983 25790 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:44.654273 25790 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:44.660267 25790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d2cfed157394b8ebdf157fa22bb456b: Generated new TSK 0
I20260619 16:50:44.660995 25790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:44.671864 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25705
W20260619 16:50:44.677255 25690 connection.cc:535] client connection to 127.3.233.190:35547 recv error: Network error: failed to read from TLS socket (remote: 127.3.233.190:35547): Connection reset by peer (error 104)
2026-06-19T16:50:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:44 dist-test-slave-cgwx krb5kdc[25804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:44 dist-test-slave-cgwx krb5kdc[25804](info): set up 2 sockets
Jun 19 16:50:44 dist-test-slave-cgwx krb5kdc[25804](info): commencing operation
krb5kdc: starting...
W20260619 16:50:45.715363 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:45 dist-test-slave-cgwx krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:45.838438 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:44179
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:45.956429 25817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:45.956739 25817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:45.956816 25817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:45.960897 25817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:45.960991 25817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:45.961032 25817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:45.961089 25817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:45.961129 25817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:45.966382 25817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:44179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:45.967590 25817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:45.968648 25817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:45.977718 25825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:45.977723 25822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:45.977907 25817 server_base.cc:1034] running on GCE node
W20260619 16:50:45.977746 25823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:45.978566 25817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:45.979835 25817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:45.981046 25817 hybrid_clock.cc:648] HybridClock initialized: now 1781887845981014 us; error 53 us; skew 500 ppm
Jun 19 16:50:45 dist-test-slave-cgwx krb5kdc[25804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887845, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:45.987859 25817 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:45.988960 25817 webserver.cc:460] Webserver started at http://127.3.233.190:43485/ using document root <none> and password file <none>
I20260619 16:50:45.989614 25817 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:45.989718 25817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:45.989979 25817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:45.992465 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005"
format_stamp: "Formatted at 2026-06-19 16:50:45 on dist-test-slave-cgwx"
server_key: "94f56735d8cda3c20a98d0cf0080c1f0"
server_key_iv: "6a758f48f8f7d26bdee933a25bc6acf5"
server_key_version: "encryptionkey@0"
I20260619 16:50:45.993103 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005"
format_stamp: "Formatted at 2026-06-19 16:50:45 on dist-test-slave-cgwx"
server_key: "94f56735d8cda3c20a98d0cf0080c1f0"
server_key_iv: "6a758f48f8f7d26bdee933a25bc6acf5"
server_key_version: "encryptionkey@0"
I20260619 16:50:45.997272 25817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:45.999964 25832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:46.001374 25817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:46.001506 25817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005"
format_stamp: "Formatted at 2026-06-19 16:50:45 on dist-test-slave-cgwx"
server_key: "94f56735d8cda3c20a98d0cf0080c1f0"
server_key_iv: "6a758f48f8f7d26bdee933a25bc6acf5"
server_key_version: "encryptionkey@0"
I20260619 16:50:46.001677 25817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:46.006589 25817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:46.009953 25817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:46.010174 25817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:46.017550 25817 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:44179
I20260619 16:50:46.017638 25884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:44179 every 8 connection(s)
I20260619 16:50:46.018631 25817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:46.021803 25885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:46.025192 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25817
I20260619 16:50:46.025353 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:46.025627 4006 external_mini_cluster.cc:1428] Setting key bedf4d1ff2e789e820b2fae52aaaebda
I20260619 16:50:46.028594 25885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005: Bootstrap starting.
I20260619 16:50:46.031633 25885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:46.031960 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:46.027236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51679 (local address 127.3.233.190:44179)
0619 16:50:46.027691 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:46.027712 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:46.027759 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:50:46.028666 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:46.028692 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:46.029091 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:46.029493 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:46.030322 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:46.030328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:46.030346 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:46.030358 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:46.030427 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:46.030454 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:46.030598 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:46.030757 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 16:50:46.030956 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260619 16:50:46.032613 25885 log.cc:824] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:46.035066 25885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005: No bootstrap required, opened a new log
I20260619 16:50:46.035192 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:46.039340 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:46.037548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34297 (local address 127.3.233.190:44179)
0619 16:50:46.037681 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:46.037686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:46.037795 (+ 109us) server_negotiation.cc:407] Connection header received
0619 16:50:46.037963 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:46.037967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:46.038016 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:46.038103 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:46.038911 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:46.038915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:46.038917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:46.038954 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:46.038988 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:46.039001 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:46.039078 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:46.039165 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:50:46.039207 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:50:46.039803 25885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44179 } }
I20260619 16:50:46.040055 25885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:46.040127 25885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1f3c5e7d9b4e7882bb4560fcdc5005, State: Initialized, Role: FOLLOWER
I20260619 16:50:46.040694 25885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44179 } }
I20260619 16:50:46.040838 25885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:46.040915 25885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:46.041033 25885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:46.042052 25885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44179 } }
I20260619 16:50:46.042433 25885 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c1f3c5e7d9b4e7882bb4560fcdc5005; no voters:
I20260619 16:50:46.042773 25885 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:46.042902 25895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:46.043200 25895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [term 1 LEADER]: Becoming Leader. State: Replica: 9c1f3c5e7d9b4e7882bb4560fcdc5005, State: Running, Role: LEADER
I20260619 16:50:46.043557 25895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44179 } }
I20260619 16:50:46.044353 25885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:46.045411 25897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1f3c5e7d9b4e7882bb4560fcdc5005. Latest consensus state: current_term: 1 leader_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44179 } } }
I20260619 16:50:46.045411 25896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c1f3c5e7d9b4e7882bb4560fcdc5005" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 44179 } } }
I20260619 16:50:46.045580 25897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:46.045593 25896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:46.045953 25904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:46.049142 25904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:46.055205 25904 catalog_manager.cc:1347] Generated new cluster ID: e3723273cdfe42c2960ba49d2b2955be
I20260619 16:50:46.055295 25904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:46.064188 25904 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:46.064571 25904 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:46.069093 25904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c1f3c5e7d9b4e7882bb4560fcdc5005: Generated new TSK 0
I20260619 16:50:46.070019 25904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:46.076499 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25817
2026-06-19T16:50:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:46 dist-test-slave-cgwx krb5kdc[25916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:46 dist-test-slave-cgwx krb5kdc[25916](info): set up 2 sockets
Jun 19 16:50:46 dist-test-slave-cgwx krb5kdc[25916](info): commencing operation
krb5kdc: starting...
W20260619 16:50:47.131276 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:47 dist-test-slave-cgwx krb5kdc[25916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:47.251436 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38355
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:33295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:38355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:47.362112 25929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:47.362465 25929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:47.362548 25929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:47.366676 25929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:47.366791 25929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:47.366827 25929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:47.366864 25929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:47.366894 25929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:47.372366 25929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:33295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:38355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:38355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:47.373514 25929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:47.374600 25929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:47.383252 25934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:47.383250 25935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:47.383592 25929 server_base.cc:1034] running on GCE node
W20260619 16:50:47.383263 25937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:47.384176 25929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:47.385311 25929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:47.386509 25929 hybrid_clock.cc:648] HybridClock initialized: now 1781887847386474 us; error 53 us; skew 500 ppm
Jun 19 16:50:47 dist-test-slave-cgwx krb5kdc[25916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887847, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:47.389362 25929 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:47.390295 25929 webserver.cc:460] Webserver started at http://127.3.233.190:35713/ using document root <none> and password file <none>
I20260619 16:50:47.390857 25929 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:47.390965 25929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:47.391209 25929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:47.393231 25929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "222cf3e2e54b4649b51e936b952eedee"
format_stamp: "Formatted at 2026-06-19 16:50:47 on dist-test-slave-cgwx"
server_key: "eb4f80219e15c31634b4e3ef1dda911a"
server_key_iv: "bca7a98a3888130487be87bd148bea3e"
server_key_version: "encryptionkey@0"
I20260619 16:50:47.393795 25929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "222cf3e2e54b4649b51e936b952eedee"
format_stamp: "Formatted at 2026-06-19 16:50:47 on dist-test-slave-cgwx"
server_key: "eb4f80219e15c31634b4e3ef1dda911a"
server_key_iv: "bca7a98a3888130487be87bd148bea3e"
server_key_version: "encryptionkey@0"
I20260619 16:50:47.397611 25929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260619 16:50:47.400097 25944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:47.401404 25929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:47.401551 25929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "222cf3e2e54b4649b51e936b952eedee"
format_stamp: "Formatted at 2026-06-19 16:50:47 on dist-test-slave-cgwx"
server_key: "eb4f80219e15c31634b4e3ef1dda911a"
server_key_iv: "bca7a98a3888130487be87bd148bea3e"
server_key_version: "encryptionkey@0"
I20260619 16:50:47.401676 25929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:47.416739 25929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:47.420459 25929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:47.420692 25929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:47.428260 25929 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:38355
I20260619 16:50:47.428275 25996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:38355 every 8 connection(s)
I20260619 16:50:47.429462 25929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:47.432412 25997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:47.438007 25997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee: Bootstrap starting.
I20260619 16:50:47.438186 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 25929
I20260619 16:50:47.438364 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:47.438642 4006 external_mini_cluster.cc:1428] Setting key c165aa0bb43fe93c1e9ec9c537f0bb30
I20260619 16:50:47.440868 25997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:47.441764 25997 log.cc:824] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:47.443861 25997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee: No bootstrap required, opened a new log
I20260619 16:50:47.447607 25997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222cf3e2e54b4649b51e936b952eedee" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38355 } }
I20260619 16:50:47.447865 25997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:47.447956 25997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222cf3e2e54b4649b51e936b952eedee, State: Initialized, Role: FOLLOWER
Jun 19 16:50:47 dist-test-slave-cgwx krb5kdc[25916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:47.448457 25997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222cf3e2e54b4649b51e936b952eedee" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38355 } }
I20260619 16:50:47.448612 25997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:47.448700 25997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:47.448808 25997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:47.450148 25997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222cf3e2e54b4649b51e936b952eedee" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38355 } }
I20260619 16:50:47.450590 25997 leader_election.cc:302] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 222cf3e2e54b4649b51e936b952eedee; no voters:
I20260619 16:50:47.450994 25997 leader_election.cc:288] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:47.451150 26002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:47.451412 26002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [term 1 LEADER]: Becoming Leader. State: Replica: 222cf3e2e54b4649b51e936b952eedee, State: Running, Role: LEADER
I20260619 16:50:47.451798 26002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222cf3e2e54b4649b51e936b952eedee" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38355 } }
I20260619 16:50:47.452466 25997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:47.453603 26004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222cf3e2e54b4649b51e936b952eedee. Latest consensus state: current_term: 1 leader_uuid: "222cf3e2e54b4649b51e936b952eedee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222cf3e2e54b4649b51e936b952eedee" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38355 } } }
I20260619 16:50:47.453994 26004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:47.453887 26003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "222cf3e2e54b4649b51e936b952eedee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222cf3e2e54b4649b51e936b952eedee" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 38355 } } }
I20260619 16:50:47.454330 26003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:47.455163 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:47.440062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37577 (local address 127.3.233.190:38355)
0619 16:50:47.440513 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:47.440531 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:47.440585 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:50:47.441454 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:47.441477 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:47.441938 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:47.442326 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:47.443221 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:47.444285 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:47.445188 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:47.445696 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:47.445867 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:47.448895 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:47.448901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:47.448918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:47.448953 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:47.451124 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:47.451685 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:47.451690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:47.451696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:47.451837 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:47.452152 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:47.452155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:47.452157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:47.452738 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:47.452939 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:47.453745 (+ 806us) server_negotiation.cc:299] Negotiation successful
0619 16:50:47.454056 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260619 16:50:47.455302 26012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:47.458429 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:47.458472 26012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:47.465174 26012 catalog_manager.cc:1347] Generated new cluster ID: cfdb5341c0664d15b4540fad0c95ae2a
I20260619 16:50:47.465255 26012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:47.469785 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:47.460790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39047 (local address 127.3.233.190:38355)
0619 16:50:47.460930 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:47.460934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:47.461331 (+ 397us) server_negotiation.cc:407] Connection header received
0619 16:50:47.461439 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:47.461442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:47.461493 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:47.461589 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:47.462430 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:47.463266 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:47.464463 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:47.464788 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:47.464889 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:47.466462 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:47.466466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:47.466469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:47.466501 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:47.468092 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:47.468534 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:47.468540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:47.468542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:47.468626 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:47.468920 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:47.468924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:47.468926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:47.469108 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:47.469243 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:47.469487 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:50:47.469604 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:50:47.485677 26012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:47.486759 26012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:47.494236 26012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 222cf3e2e54b4649b51e936b952eedee: Generated new TSK 0
I20260619 16:50:47.495167 26012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:47.499630 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 25929
2026-06-19T16:50:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:47 dist-test-slave-cgwx krb5kdc[26028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:47 dist-test-slave-cgwx krb5kdc[26028](info): set up 2 sockets
Jun 19 16:50:47 dist-test-slave-cgwx krb5kdc[26028](info): commencing operation
krb5kdc: starting...
W20260619 16:50:48.567610 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:48 dist-test-slave-cgwx krb5kdc[26028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:48.692122 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:43067
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:43067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:48.803704 26041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:48.804034 26041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:48.804111 26041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:48.808252 26041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:48.808343 26041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:48.808382 26041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:48.808439 26041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:48.808477 26041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:48.813830 26041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:43067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:48.815029 26041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:48.816067 26041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:48.824645 26049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:48.824640 26046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:48.824640 26047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:48.824776 26041 server_base.cc:1034] running on GCE node
I20260619 16:50:48.825413 26041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:48.826532 26041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:48.827731 26041 hybrid_clock.cc:648] HybridClock initialized: now 1781887848827700 us; error 45 us; skew 500 ppm
Jun 19 16:50:48 dist-test-slave-cgwx krb5kdc[26028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887848, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:48.830698 26041 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:48.831730 26041 webserver.cc:460] Webserver started at http://127.3.233.190:37677/ using document root <none> and password file <none>
I20260619 16:50:48.832325 26041 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:48.832417 26041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:48.832649 26041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:48.834687 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "4148de1473344014b3b8dd7f6d2ef94b"
format_stamp: "Formatted at 2026-06-19 16:50:48 on dist-test-slave-cgwx"
server_key: "4466d8a1107899e241e2cb2a8359439e"
server_key_iv: "f89e552fec3cd01ca82fe63c6e6b3578"
server_key_version: "encryptionkey@0"
I20260619 16:50:48.835289 26041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "4148de1473344014b3b8dd7f6d2ef94b"
format_stamp: "Formatted at 2026-06-19 16:50:48 on dist-test-slave-cgwx"
server_key: "4466d8a1107899e241e2cb2a8359439e"
server_key_iv: "f89e552fec3cd01ca82fe63c6e6b3578"
server_key_version: "encryptionkey@0"
I20260619 16:50:48.839141 26041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:50:48.841604 26056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:48.843116 26041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:48.843304 26041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "4148de1473344014b3b8dd7f6d2ef94b"
format_stamp: "Formatted at 2026-06-19 16:50:48 on dist-test-slave-cgwx"
server_key: "4466d8a1107899e241e2cb2a8359439e"
server_key_iv: "f89e552fec3cd01ca82fe63c6e6b3578"
server_key_version: "encryptionkey@0"
I20260619 16:50:48.843425 26041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:48.861302 26041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:48.864893 26041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:48.865113 26041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:48.872355 26041 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:43067
I20260619 16:50:48.872382 26108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:43067 every 8 connection(s)
I20260619 16:50:48.873332 26041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:48.876243 26109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:48.878823 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26041
I20260619 16:50:48.879025 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:48.879333 4006 external_mini_cluster.cc:1428] Setting key 6e4cf28b3a52b3c86bc8e100a97369b4
I20260619 16:50:48.882633 26109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b: Bootstrap starting.
I20260619 16:50:48.884927 26109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:48.885877 26109 log.cc:824] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:48.888069 26109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b: No bootstrap required, opened a new log
I20260619 16:50:48.888823 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:48.880844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53931 (local address 127.3.233.190:43067)
0619 16:50:48.881350 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:48.881365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:48.881407 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:50:48.882071 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:48.882090 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:48.882434 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:48.882857 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:48.883295 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:48.884250 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:48.885176 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:48.885880 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:48.886066 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:48.887071 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:48.887076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:48.887089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:48.887467 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:48.887519 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:48.887598 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:48.887715 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:48.887831 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:50:48.888047 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":140,"threads_started":1}
I20260619 16:50:48.892051 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:48.892436 26109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4148de1473344014b3b8dd7f6d2ef94b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43067 } }
I20260619 16:50:48.892697 26109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:48.892756 26109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4148de1473344014b3b8dd7f6d2ef94b, State: Initialized, Role: FOLLOWER
I20260619 16:50:48.893225 26109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4148de1473344014b3b8dd7f6d2ef94b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43067 } }
I20260619 16:50:48.893392 26109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:48.893471 26109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:48.893576 26109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:48.894742 26109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4148de1473344014b3b8dd7f6d2ef94b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43067 } }
I20260619 16:50:48.895154 26109 leader_election.cc:302] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4148de1473344014b3b8dd7f6d2ef94b; no voters:
I20260619 16:50:48.895498 26109 leader_election.cc:288] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:48.895578 26119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:48.895788 26119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [term 1 LEADER]: Becoming Leader. State: Replica: 4148de1473344014b3b8dd7f6d2ef94b, State: Running, Role: LEADER
I20260619 16:50:48.896157 26119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4148de1473344014b3b8dd7f6d2ef94b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43067 } }
I20260619 16:50:48.896844 26109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:48.897861 26120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4148de1473344014b3b8dd7f6d2ef94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4148de1473344014b3b8dd7f6d2ef94b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43067 } } }
I20260619 16:50:48.897991 26120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:48.899390 26121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4148de1473344014b3b8dd7f6d2ef94b. Latest consensus state: current_term: 1 leader_uuid: "4148de1473344014b3b8dd7f6d2ef94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4148de1473344014b3b8dd7f6d2ef94b" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 43067 } } }
I20260619 16:50:48.899835 26121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:48.899570 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:48.894266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40401 (local address 127.3.233.190:43067)
0619 16:50:48.894400 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:48.894405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:48.894613 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:50:48.894790 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:48.894793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:48.894836 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:48.895013 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:48.895437 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:48.896214 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:48.897176 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:48.897518 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:48.897697 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:48.898772 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:48.898776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:48.898778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:48.898791 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:48.898823 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:48.898840 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:48.898913 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:48.899295 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 16:50:48.899364 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:50:48.902621 26136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:48.902704 26136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:48.902769 26128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:48.903558 26128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:48.909150 26128 catalog_manager.cc:1347] Generated new cluster ID: f14be927172644859b502d8dd15edd94
I20260619 16:50:48.909260 26128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:48.924814 26128 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:48.926208 26128 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:48.934374 26128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4148de1473344014b3b8dd7f6d2ef94b: Generated new TSK 0
I20260619 16:50:48.935317 26128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:48.944016 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26041
2026-06-19T16:50:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:49 dist-test-slave-cgwx krb5kdc[26141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:49 dist-test-slave-cgwx krb5kdc[26141](info): set up 2 sockets
Jun 19 16:50:49 dist-test-slave-cgwx krb5kdc[26141](info): commencing operation
krb5kdc: starting...
W20260619 16:50:50.035241 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:50 dist-test-slave-cgwx krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:50.157814 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37789
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:37789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:50.278179 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:50.278545 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:50.278656 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:50.283399 26154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:50.283509 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:50.283552 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:50.283617 26154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:50.283658 26154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:50.289778 26154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:37789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:37789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:50.290993 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:50.292110 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:50.301082 26159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:50.301097 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:50.301103 26160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:50.301455 26154 server_base.cc:1034] running on GCE node
I20260619 16:50:50.302068 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:50.303474 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:50.304802 26154 hybrid_clock.cc:648] HybridClock initialized: now 1781887850304708 us; error 41 us; skew 500 ppm
Jun 19 16:50:50 dist-test-slave-cgwx krb5kdc[26141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:50.308138 26154 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:50.309134 26154 webserver.cc:460] Webserver started at http://127.3.233.190:42723/ using document root <none> and password file <none>
I20260619 16:50:50.309731 26154 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:50.309825 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:50.310060 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:50.312232 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e"
format_stamp: "Formatted at 2026-06-19 16:50:50 on dist-test-slave-cgwx"
server_key: "f4d5c242ab70e717c867dc4289b274ce"
server_key_iv: "d314230062cc65abbe278589e64f2dec"
server_key_version: "encryptionkey@0"
I20260619 16:50:50.312798 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e"
format_stamp: "Formatted at 2026-06-19 16:50:50 on dist-test-slave-cgwx"
server_key: "f4d5c242ab70e717c867dc4289b274ce"
server_key_iv: "d314230062cc65abbe278589e64f2dec"
server_key_version: "encryptionkey@0"
I20260619 16:50:50.316658 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:50:50.319247 26169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:50.320506 26154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:50:50.320652 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e"
format_stamp: "Formatted at 2026-06-19 16:50:50 on dist-test-slave-cgwx"
server_key: "f4d5c242ab70e717c867dc4289b274ce"
server_key_iv: "d314230062cc65abbe278589e64f2dec"
server_key_version: "encryptionkey@0"
I20260619 16:50:50.320815 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:50.325558 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:50.329200 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:50.329422 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:50.336836 26154 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:37789
I20260619 16:50:50.336850 26221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:37789 every 8 connection(s)
I20260619 16:50:50.337807 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:50.340672 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:50.344465 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26154
I20260619 16:50:50.344651 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:50.344923 4006 external_mini_cluster.cc:1428] Setting key deffe868815acd3de24df668a3985ee4
I20260619 16:50:50.346467 26222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e: Bootstrap starting.
I20260619 16:50:50.348973 26222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:50.349845 26222 log.cc:824] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:50.350598 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:50.346394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37263 (local address 127.3.233.190:37789)
0619 16:50:50.346851 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:50.346867 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:50.346902 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:50:50.347626 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:50.347644 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:50.347944 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:50.348228 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:50.349151 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:50.349155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:50.349170 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:50.349183 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:50.349239 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:50.349258 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:50.349346 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:50.349486 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 16:50:50.349658 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":156,"threads_started":1}
I20260619 16:50:50.351871 26222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e: No bootstrap required, opened a new log
I20260619 16:50:50.353391 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:50.355780 26222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37789 } }
I20260619 16:50:50.356096 26222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:50.356209 26222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be8c4ee0a8e49649b5b76ea4ecc076e, State: Initialized, Role: FOLLOWER
I20260619 16:50:50.356730 26222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37789 } }
I20260619 16:50:50.356904 26222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:50.356977 26222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:50.357093 26222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:50.357311 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:50.355514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34749 (local address 127.3.233.190:37789)
0619 16:50:50.355649 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:50.355656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:50.355876 (+ 220us) server_negotiation.cc:407] Connection header received
0619 16:50:50.355960 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:50.355963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:50.355999 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:50.356095 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:50.356877 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:50.356880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:50.356883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:50:50.356894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:50.356924 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:50:50.356937 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:50.357036 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:50.357145 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 16:50:50.357190 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:50:50.358239 26222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37789 } }
I20260619 16:50:50.358582 26222 leader_election.cc:302] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0be8c4ee0a8e49649b5b76ea4ecc076e; no voters:
I20260619 16:50:50.358865 26222 leader_election.cc:288] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:50.358989 26232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:50.359246 26232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [term 1 LEADER]: Becoming Leader. State: Replica: 0be8c4ee0a8e49649b5b76ea4ecc076e, State: Running, Role: LEADER
I20260619 16:50:50.359591 26232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37789 } }
I20260619 16:50:50.360133 26222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:50.361431 26234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0be8c4ee0a8e49649b5b76ea4ecc076e. Latest consensus state: current_term: 1 leader_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37789 } } }
I20260619 16:50:50.361462 26233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be8c4ee0a8e49649b5b76ea4ecc076e" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 37789 } } }
I20260619 16:50:50.361626 26233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:50.361619 26234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:50.362025 26242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:50.365224 26242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:50.370620 26242 catalog_manager.cc:1347] Generated new cluster ID: a0eeb9ee084d4fd5a364e267d4d9cbc7
I20260619 16:50:50.370704 26242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:50.406301 26242 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:50.406677 26242 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:50.415290 26242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0be8c4ee0a8e49649b5b76ea4ecc076e: Generated new TSK 0
I20260619 16:50:50.416021 26242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:50:50.423801 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26154
2026-06-19T16:50:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1479 ms)
[----------] 18 tests from AuthTokenIssuingTest (25932 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:50 dist-test-slave-cgwx krb5kdc[26253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:50 dist-test-slave-cgwx krb5kdc[26253](info): set up 2 sockets
Jun 19 16:50:50 dist-test-slave-cgwx krb5kdc[26253](info): commencing operation
krb5kdc: starting...
W20260619 16:50:51.475625 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:51 dist-test-slave-cgwx krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:51.602466 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:33301
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:33301,127.3.233.189:34847,127.3.233.188:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:51.719400 26266 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:50:51.719697 26266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:51.719720 26266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:51.719813 26266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:51.723889 26266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:51.723958 26266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:51.723985 26266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:51.724002 26266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:51.724062 26266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:51.729460 26266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:33301,127.3.233.189:34847,127.3.233.188:42479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:51.730665 26266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:51.731727 26266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:51.740372 26272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:51.740355 26271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:51.740523 26266 server_base.cc:1034] running on GCE node
W20260619 16:50:51.740329 26274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:51.741300 26266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:51.742472 26266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:51.743659 26266 hybrid_clock.cc:648] HybridClock initialized: now 1781887851743641 us; error 31 us; skew 500 ppm
Jun 19 16:50:51 dist-test-slave-cgwx krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:51.753677 26266 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:51.754637 26266 webserver.cc:460] Webserver started at http://127.3.233.190:33565/ using document root <none> and password file <none>
I20260619 16:50:51.755244 26266 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:51.755308 26266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:51.755477 26266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:51.757539 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c"
format_stamp: "Formatted at 2026-06-19 16:50:51 on dist-test-slave-cgwx"
server_key: "03f8f0ff6d2c40f6eda3b47c1829a5d0"
server_key_iv: "a64eb1ede7554d20a475edc18fefab35"
server_key_version: "encryptionkey@0"
I20260619 16:50:51.758082 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c"
format_stamp: "Formatted at 2026-06-19 16:50:51 on dist-test-slave-cgwx"
server_key: "03f8f0ff6d2c40f6eda3b47c1829a5d0"
server_key_iv: "a64eb1ede7554d20a475edc18fefab35"
server_key_version: "encryptionkey@0"
I20260619 16:50:51.761907 26266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:50:51.764505 26281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:51.765755 26266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:51.765865 26266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c"
format_stamp: "Formatted at 2026-06-19 16:50:51 on dist-test-slave-cgwx"
server_key: "03f8f0ff6d2c40f6eda3b47c1829a5d0"
server_key_iv: "a64eb1ede7554d20a475edc18fefab35"
server_key_version: "encryptionkey@0"
I20260619 16:50:51.765954 26266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:51.784895 26266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:51.788113 26266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:51.788286 26266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:51.795467 26266 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:33301
I20260619 16:50:51.795804 26333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:33301 every 8 connection(s)
I20260619 16:50:51.796479 26266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:51.799228 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26266
I20260619 16:50:51.799381 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:51.799710 4006 external_mini_cluster.cc:1428] Setting key 29d2dad547066adcc7899e5632038ffa
I20260619 16:50:51.799849 26334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:51.804550 26334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.233.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.189@KRBTEST.COM" created.
Jun 19 16:50:51 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:51.821545 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.807001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33301 (local address 127.0.0.1:54826)
0619 16:50:51.807536 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:51.807595 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:51.808723 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:51.809518 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:51.809523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:51.809964 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:51.810346 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:51.810369 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:51.811808 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:51.811811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:51.812622 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:51.812633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:51.813692 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:51.813700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:51.813822 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:51.814423 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:51.814439 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:51.816186 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:51.818469 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:51.818477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:51.818486 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:51.818782 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:51.819144 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:51.819146 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:51.819147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:51.819265 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:51.819976 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:51.819985 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:51.820176 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 16:50:51.820331 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 16:50:51.820730 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":100,"thread_start_us":164,"threads_started":1}
I20260619 16:50:51.821578 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.808006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.3.233.190:33301)
0619 16:50:51.808477 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:51.808486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:51.808525 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:50:51.809042 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:51.809049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:51.809303 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:51.809464 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:51.810524 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:51.811662 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:51.812751 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:51.813438 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:51.813625 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:51.816358 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:51.816365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:51.816382 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:51.816414 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:51.818361 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:51.818884 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:51.818889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:51.818890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:51.819050 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:51.819389 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:51.819392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:51.819393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:51.819856 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:51.819964 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:51.820416 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 16:50:51.820660 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":105,"threads_started":1}
I20260619 16:50:51.824277 26334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab.
Entry for principal kudu/127.3.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab.
I20260619 16:50:51.825112 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.824791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35564)
0619 16:50:51.824937 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:51.825022 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:50:51.825335 26284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
I20260619 16:50:51.826004 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:34847
--webserver_interface=127.3.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab
--principal=kudu/127.3.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:33301,127.3.233.189:34847,127.3.233.188:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:50:51.826233 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.826007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35576)
0619 16:50:51.826115 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:51.826164 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:50:51.826395 26334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:34847: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
I20260619 16:50:51.875694 26334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } attempt: 1
I20260619 16:50:51.876605 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.876282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35588)
0619 16:50:51.876447 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:51.876516 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:50:51.877430 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.877190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35594)
0619 16:50:51.877319 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:51.877370 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 16:50:51.877605 26334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:34847: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
W20260619 16:50:51.937163 26341 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:50:51.937449 26341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:51.937474 26341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:51.937567 26341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:51.941691 26341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:51.941778 26341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:51.941798 26341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:51.941816 26341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:51.941859 26341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:51.945777 26334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } attempt: 2
I20260619 16:50:51.946700 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.946349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35608)
0619 16:50:51.946556 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:51.946624 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:50:51.947417 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:51.947227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35622)
0619 16:50:51.947319 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:51.947364 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 16:50:51.947582 26334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:34847: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34847: connect: Connection refused (error 111)
I20260619 16:50:51.947695 26341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:33301,127.3.233.189:34847,127.3.233.188:42479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab
--principal=kudu/127.3.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:51.948781 26341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:51.949787 26341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:51.958541 26347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:51.958541 26346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:51.958564 26349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:51.958665 26341 server_base.cc:1034] running on GCE node
I20260619 16:50:51.959510 26341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:51.960632 26341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:51.961843 26341 hybrid_clock.cc:648] HybridClock initialized: now 1781887851961828 us; error 28 us; skew 500 ppm
Jun 19 16:50:51 dist-test-slave-cgwx krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:51.964649 26341 init.cc:377] Logged in from keytab as kudu/127.3.233.189@KRBTEST.COM (short username kudu)
I20260619 16:50:51.965585 26341 webserver.cc:460] Webserver started at http://127.3.233.189:43943/ using document root <none> and password file <none>
I20260619 16:50:51.966143 26341 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:51.966228 26341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:51.966454 26341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:51.968562 26341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data/instance:
uuid: "3333eb9f5f6c4287ace8963b667c1c54"
format_stamp: "Formatted at 2026-06-19 16:50:51 on dist-test-slave-cgwx"
server_key: "3ece3d03a5e689ef8a2a2e4c2e425c38"
server_key_iv: "55908ae73c5d2d45ecbe519bbbf5d8b1"
server_key_version: "encryptionkey@0"
I20260619 16:50:51.969154 26341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/wal/instance:
uuid: "3333eb9f5f6c4287ace8963b667c1c54"
format_stamp: "Formatted at 2026-06-19 16:50:51 on dist-test-slave-cgwx"
server_key: "3ece3d03a5e689ef8a2a2e4c2e425c38"
server_key_iv: "55908ae73c5d2d45ecbe519bbbf5d8b1"
server_key_version: "encryptionkey@0"
I20260619 16:50:51.973026 26341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 16:50:51.975530 26356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:51.977005 26341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:51.977147 26341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/wal
uuid: "3333eb9f5f6c4287ace8963b667c1c54"
format_stamp: "Formatted at 2026-06-19 16:50:51 on dist-test-slave-cgwx"
server_key: "3ece3d03a5e689ef8a2a2e4c2e425c38"
server_key_iv: "55908ae73c5d2d45ecbe519bbbf5d8b1"
server_key_version: "encryptionkey@0"
I20260619 16:50:51.977288 26341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:52.020509 26341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:52.024065 26341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:52.024268 26341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:52.031848 26341 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.189:34847
I20260619 16:50:52.032233 26408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.189:34847 every 8 connection(s)
I20260619 16:50:52.032897 26341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
I20260619 16:50:52.035934 26409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:52.038779 26334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } attempt: 3
I20260619 16:50:52.040390 26409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:52.043227 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26341
I20260619 16:50:52.043360 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-1/wal/instance
I20260619 16:50:52.043675 4006 external_mini_cluster.cc:1428] Setting key 14e417298fcca3c5a000046604687612
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.233.188@KRBTEST.COM; defaulting to no policy
I20260619 16:50:52.054317 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.039334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35628)
0619 16:50:52.039487 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.039504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.039608 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.041989 (+ 2381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.041992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.042221 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.042360 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.042367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.044040 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.044046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.044869 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.044876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.046090 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.046095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.046210 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.046828 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.046844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.048435 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.050961 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.050967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.050969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.051277 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.052627 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.052630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.052632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.052740 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.053689 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.053692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.053837 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.053946 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.054109 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:50:52.054464 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.042225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.3.233.190:33301)
0619 16:50:52.042338 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.042342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.042741 (+ 399us) server_negotiation.cc:407] Connection header received
0619 16:50:52.042870 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.042872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.042958 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.043048 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.044064 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.044869 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.046002 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.046987 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.047120 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.050248 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.050255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.050258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.050291 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.052224 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.052902 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.052906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.052908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.053024 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.053310 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.053313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.053315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.053516 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.053681 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.054099 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.054245 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.188@KRBTEST.COM" created.
I20260619 16:50:52.055426 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.042223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33301 (local address 127.0.0.1:54838)
0619 16:50:52.042602 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.042653 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.042780 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.043114 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.043122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.043627 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.043861 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.043876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.045010 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.045017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.045871 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.045881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.047159 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.047166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.047314 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.048006 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.048043 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.050087 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.052406 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.052418 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.052425 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.052771 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.053128 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.053130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.053131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.053204 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.053701 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.053708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.053860 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.054039 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.054427 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":33,"thread_start_us":119,"threads_started":1}
I20260619 16:50:52.055424 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.040127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.3.233.189:34847)
0619 16:50:52.040626 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.040643 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.040681 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:50:52.041382 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.041400 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.041738 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.042045 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.042598 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.043902 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.045015 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.045924 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.046151 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.048591 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.048597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.048608 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.048630 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.050775 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.051424 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.051434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.051443 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.052496 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.052902 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.052911 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.052913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.053553 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.053689 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.054098 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.054331 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":317,"thread_start_us":157,"threads_started":1}
I20260619 16:50:52.057224 26409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:52.058267 26334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:52.059123 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.058807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35938)
0619 16:50:52.058960 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.059027 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:50:52.059865 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.059572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35940)
0619 16:50:52.059766 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.059814 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260619 16:50:52.060014 26334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:42479: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:50:52.066383 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.057755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35644)
0619 16:50:52.057908 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.057928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.058018 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.058267 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.058273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.058475 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.058618 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.058629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.060185 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.060187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.060707 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.060714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.061369 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.061371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.061442 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.061959 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.061970 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.063212 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.064954 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.064957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.064959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.065144 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.065405 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.065407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.065408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.065482 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.065968 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.065971 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.066051 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.066151 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.066262 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:50:52.066742 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.057867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.3.233.189:34847)
0619 16:50:52.057981 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.057985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.057998 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:50:52.058101 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.058104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.058151 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.058236 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.058806 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.060077 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.060833 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.061258 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.061387 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.063353 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.063359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.063364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.063396 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.064860 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.065233 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.065236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.065237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.065304 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.065558 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.065560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.065561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.065851 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.065961 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.066232 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.066332 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:50:52.067615 26409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:52.068310 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.068069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35950)
0619 16:50:52.068180 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.068261 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:50:52.068938 26357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
I20260619 16:50:52.070057 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.069814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35954)
0619 16:50:52.069910 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.069978 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 16:50:52.070221 26409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:42479: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab.
Entry for principal kudu/127.3.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab.
I20260619 16:50:52.073583 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:42479
--webserver_interface=127.3.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab
--principal=kudu/127.3.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:33301,127.3.233.189:34847,127.3.233.188:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:50:52.091194 26334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } attempt: 1
I20260619 16:50:52.092191 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.091819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35964)
0619 16:50:52.092004 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.092082 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:50:52.093056 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.092792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35970)
0619 16:50:52.092906 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.092965 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:50:52.093248 26334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:42479: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
I20260619 16:50:52.119549 26409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } attempt: 1
I20260619 16:50:52.120460 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.120142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35976)
0619 16:50:52.120316 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.120386 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:50:52.121243 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.121026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:35992)
0619 16:50:52.121131 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.121181 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:50:52.121415 26409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:42479: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
I20260619 16:50:52.168440 26334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } attempt: 2
I20260619 16:50:52.169420 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.169022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:36008)
0619 16:50:52.169240 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.169322 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:50:52.170261 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.169994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:36012)
0619 16:50:52.170117 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.170193 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:50:52.170425 26334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:42479: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
W20260619 16:50:52.188597 26416 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:50:52.188922 26416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:52.188946 26416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:52.188987 26416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:50:52.189610 26409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } attempt: 2
I20260619 16:50:52.190531 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.190201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:36014)
0619 16:50:52.190365 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.190434 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 16:50:52.191453 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.191205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:36016)
0619 16:50:52.191345 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.191398 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 16:50:52.191690 26409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:42479: Network error: Client connection negotiation failed: client connection to 127.3.233.188:42479: connect: Connection refused (error 111)
W20260619 16:50:52.193643 26416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:52.193704 26416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:52.193725 26416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:52.193779 26416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:52.193811 26416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:52.199532 26416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:33301,127.3.233.189:34847,127.3.233.188:42479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab
--principal=kudu/127.3.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:52.200691 26416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:52.201771 26416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:52.211239 26422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:52.215791 26421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:52.215791 26424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:52.216593 26416 server_base.cc:1034] running on GCE node
I20260619 16:50:52.217358 26416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:52.218550 26416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:52.219750 26416 hybrid_clock.cc:648] HybridClock initialized: now 1781887852219706 us; error 58 us; skew 500 ppm
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:52.222862 26416 init.cc:377] Logged in from keytab as kudu/127.3.233.188@KRBTEST.COM (short username kudu)
I20260619 16:50:52.223862 26416 webserver.cc:460] Webserver started at http://127.3.233.188:36407/ using document root <none> and password file <none>
I20260619 16:50:52.224444 26416 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:52.224541 26416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:52.224774 26416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:52.226882 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data/instance:
uuid: "fec6947d68df4a5bac66f4ef5b066e9a"
format_stamp: "Formatted at 2026-06-19 16:50:52 on dist-test-slave-cgwx"
server_key: "47b93120d65ec1ca398ad22dabd392bd"
server_key_iv: "d1f3a77d2ae5fd452e5a297f749c316f"
server_key_version: "encryptionkey@0"
I20260619 16:50:52.227507 26416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/wal/instance:
uuid: "fec6947d68df4a5bac66f4ef5b066e9a"
format_stamp: "Formatted at 2026-06-19 16:50:52 on dist-test-slave-cgwx"
server_key: "47b93120d65ec1ca398ad22dabd392bd"
server_key_iv: "d1f3a77d2ae5fd452e5a297f749c316f"
server_key_version: "encryptionkey@0"
I20260619 16:50:52.231433 26416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 16:50:52.234048 26431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:52.235469 26416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:50:52.235613 26416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/wal
uuid: "fec6947d68df4a5bac66f4ef5b066e9a"
format_stamp: "Formatted at 2026-06-19 16:50:52 on dist-test-slave-cgwx"
server_key: "47b93120d65ec1ca398ad22dabd392bd"
server_key_iv: "d1f3a77d2ae5fd452e5a297f749c316f"
server_key_version: "encryptionkey@0"
I20260619 16:50:52.235740 26416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:52.256016 26416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:52.259701 26416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:52.259936 26416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:52.267205 26416 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.188:42479
I20260619 16:50:52.267529 26483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.188:42479 every 8 connection(s)
I20260619 16:50:52.268255 26416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
I20260619 16:50:52.269613 26334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } attempt: 3
I20260619 16:50:52.270358 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26416
I20260619 16:50:52.270496 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887569513258-4006-0/minicluster-data/master-2/wal/instance
I20260619 16:50:52.270882 4006 external_mini_cluster.cc:1428] Setting key 6d931b0afc74ebe013a0f80781f9b897
I20260619 16:50:52.272559 26484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:52.277590 26484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:50:52.282863 26409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } attempt: 3
I20260619 16:50:52.284513 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.270160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:36026)
0619 16:50:52.270373 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.270390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.270487 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.273270 (+ 2783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.273273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.273498 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.273626 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.273635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.275308 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.275312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.276125 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.276137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.276763 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.276766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.276844 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.277408 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.277423 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.279055 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.281951 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.281957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.281962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.282244 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.282661 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.282665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.282667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.282773 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.283856 (+ 1083us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.283862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.283981 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.284103 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.284263 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 16:50:52.286240 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.271504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.3.233.188:42479)
0619 16:50:52.272195 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.272201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.272212 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:50:52.272536 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.272544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.272962 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.273317 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.274292 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.275162 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.276101 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.276576 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.276750 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.280256 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.280260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.280263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.280290 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.282478 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.283001 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.283006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.283011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.283098 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.283402 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.283406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.283407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.283738 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.283962 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.284824 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.285157 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":636,"spinlock_wait_cycles":6272,"thread_start_us":111,"threads_started":1}
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:52.286237 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.270910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.3.233.188:42479)
0619 16:50:52.271526 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.271544 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.271598 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:50:52.272521 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.272544 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.272962 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.273741 (+ 779us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.273867 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.275165 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.276271 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.276626 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.277075 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.279259 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.279265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.279327 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.279368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.281812 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.282394 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.282402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.282409 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.282558 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.283172 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.283178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.283180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.283738 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.283868 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.285072 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.285216 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":5120,"thread_start_us":147,"threads_started":1}
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:50:52.291478 26334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c: Bootstrap starting.
I20260619 16:50:52.291531 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.279730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.3.233.190:33301)
0619 16:50:52.279863 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.279869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.280346 (+ 477us) server_negotiation.cc:407] Connection header received
0619 16:50:52.280526 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.280529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.280573 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.280662 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.281539 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.282176 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.283980 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.284327 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.284427 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.287491 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.287496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.287498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.287531 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.289349 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.289889 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.289893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.289895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.289985 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.290290 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.290295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.290296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.290532 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.290625 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.291272 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.291382 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:50:52.292845 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.279703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:33301 (local address 127.0.0.1:54848)
0619 16:50:52.280142 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.280268 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.280438 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.280691 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.280697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.281217 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.281414 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.281421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.282296 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.282299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.283307 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.283319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.284550 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.284557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.284673 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.285179 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.285200 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.287310 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.289470 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.289481 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.289487 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.289774 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.290090 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.290093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.290095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.290185 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.290678 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.290697 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.290816 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.291026 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.292640 (+ 1614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":137,"threads_started":1}
I20260619 16:50:52.294165 26334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:52.294231 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.283341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:36044)
0619 16:50:52.283477 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.283493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.283566 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.284377 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.284381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.284566 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.284689 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.284698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.286495 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.286513 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.287226 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.287236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.288137 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.288141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.288246 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.288957 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.288972 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.290444 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.292634 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.292637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.292639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.292883 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.293170 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.293173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.293175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.293268 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.293688 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.293690 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.293804 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.293922 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.294059 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:50:52.294241 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.283697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.3.233.188:42479)
0619 16:50:52.284115 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.284124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.284136 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:50:52.284166 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.284168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.284211 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.284804 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.285216 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.286353 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.287345 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.287688 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.289951 (+ 2263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.290575 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.290579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.290581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.290608 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.292533 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.292996 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.292999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.293001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.293078 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.293371 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.293374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.293376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.293581 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.293695 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.293981 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.294090 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":151,"threads_started":1}
I20260619 16:50:52.297472 26409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: Bootstrap starting.
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:52.299360 26334 log.cc:824] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:52.299827 26484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:52.300164 26409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:52.301100 26409 log.cc:824] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:52.302012 26334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c: No bootstrap required, opened a new log
I20260619 16:50:52.303241 26409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: No bootstrap required, opened a new log
I20260619 16:50:52.303808 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.289632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.3.233.190:33301)
0619 16:50:52.290580 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.290603 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.290616 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:50:52.290651 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.290656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.290700 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.290796 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.292301 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.293692 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.295295 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.295541 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.295626 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.298517 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.298526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.298529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.298559 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.301303 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.301865 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.301869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.301871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.301958 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.302322 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.302325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.302328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.302538 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.302778 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.303436 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.303567 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":103,"threads_started":1}
I20260619 16:50:52.306339 26334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.306613 26334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d4adb2e76f4fa7bd4a6cbbd84c353c, State: Initialized, Role: FOLLOWER
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:50:52.307017 26334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.307152 26409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.307380 26409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3333eb9f5f6c4287ace8963b667c1c54, State: Initialized, Role: FOLLOWER
I20260619 16:50:52.307682 26409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.308507 26334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:52.308883 26409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:52.312031 26495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.312179 26495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
W20260619 16:50:52.314708 26518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:52.314844 26518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 16:50:52.315897 26514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:52.315979 26514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:52.318861 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.305842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.3.233.189:34847)
0619 16:50:52.306492 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.306512 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.306527 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:50:52.306562 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.306565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.306774 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.308056 (+ 1282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.308216 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.309245 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.310391 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.310710 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.310854 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.314094 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.314098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.314100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.314124 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.316569 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.317115 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.317118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.317120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.317218 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.317502 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.317506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.317507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.317734 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.317931 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.318557 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.318670 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":107,"threads_started":1}
I20260619 16:50:52.319799 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.301063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.3.233.189:34847)
0619 16:50:52.301198 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.301203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.301296 (+ 93us) server_negotiation.cc:407] Connection header received
0619 16:50:52.301482 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.301485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.301539 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.301593 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.302234 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.302968 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.304296 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.304606 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.304689 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.307652 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.307658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.307662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.307689 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.309524 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.310027 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.310031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.310033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.310109 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.310428 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.310431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.310433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.310656 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.310730 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.316359 (+ 5629us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.317100 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:50:52.311774 26496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.320117 26496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:52.320591 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.300783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34847 (local address 127.0.0.1:35656)
0619 16:50:52.301217 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.301236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.301371 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.301756 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.301759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.301945 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.302074 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.302085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.303109 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.303112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.304125 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.304138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.305092 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.305096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.305200 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.305760 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.305776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.307476 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.309650 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.309662 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.309664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.309915 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.310211 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.310214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.310216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.310326 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.310812 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.310817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.311015 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.320215 (+ 9200us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.320407 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350}
I20260619 16:50:52.317600 26266 master_runner.cc:186] Error getting master registration for 127.3.233.188:42479: OK, instance_id { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" instance_seqno: 1781887852265170 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:50:52.321090 26266 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:50:52.321213 26484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:52.322351 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:50:52.320079 26341 master_runner.cc:186] Error getting master registration for 127.3.233.188:42479: OK, instance_id { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" instance_seqno: 1781887852265170 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:50:52.323139 26341 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:50:52.323675 26308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54850
I20260619 16:50:52.323912 26308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:50:52.324023 26308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:52.325299 26308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.326159 26308 leader_election.cc:288] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [CANDIDATE]: Term 1 election: Requested vote from peers 3333eb9f5f6c4287ace8963b667c1c54 (127.3.233.189:34847), fec6947d68df4a5bac66f4ef5b066e9a (127.3.233.188:42479)
I20260619 16:50:52.326591 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3333eb9f5f6c4287ace8963b667c1c54"
I20260619 16:50:52.326835 26378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:52.328301 26378 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d4adb2e76f4fa7bd4a6cbbd84c353c in term 1.
I20260619 16:50:52.328719 26284 leader_election.cc:302] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d4adb2e76f4fa7bd4a6cbbd84c353c, 3333eb9f5f6c4287ace8963b667c1c54; no voters:
I20260619 16:50:52.329021 26495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:52.329270 26495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [term 1 LEADER]: Becoming Leader. State: Replica: 21d4adb2e76f4fa7bd4a6cbbd84c353c, State: Running, Role: LEADER
I20260619 16:50:52.329645 26495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.331679 26495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d4adb2e76f4fa7bd4a6cbbd84c353c. Latest consensus state: current_term: 1 leader_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.331823 26495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:52.332566 26530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:52.333905 26530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 16:50:52 dist-test-slave-cgwx krb5kdc[26253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887852, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:50:52.338755 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.328286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.3.233.190:33301)
0619 16:50:52.328409 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.328414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.328578 (+ 164us) server_negotiation.cc:407] Connection header received
0619 16:50:52.328780 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.328783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.328838 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.328895 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.329932 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.330762 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.331994 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.332349 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.332439 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.334982 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.334986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.334989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.335016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.336742 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.337253 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.337256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.337258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.337344 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.337660 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.337664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.337666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.337913 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.338047 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.338426 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.338567 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:50:52.339689 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.321773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.3.233.188:42479)
0619 16:50:52.323420 (+ 1647us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.323425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.323574 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:50:52.323700 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.323706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.323755 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.323842 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.324328 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.325070 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.326174 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.332231 (+ 6057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.332363 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.335208 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.335213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.335216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.335237 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.337452 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.338109 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.338124 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.338126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.338218 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.338623 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.338626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.338628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.338826 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.338903 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.339380 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.339504 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1560}
I20260619 16:50:52.340255 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.321700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:42479 (local address 127.0.0.1:36050)
0619 16:50:52.323491 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:52.323505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:52.323598 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:52.323887 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:52.323890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:52.324079 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:52.324186 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.324193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.325193 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.325197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.326044 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:52.326055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.332485 (+ 6430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.332489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:52.332573 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.333317 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:52.333332 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:52.335058 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:52.337572 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.337577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.337582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.337962 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.338366 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:52.338369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:52.338371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:52.338497 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:52.339035 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:52.339038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:52.339166 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 16:50:52.339840 (+ 674us) client_negotiation.cc:240] Negotiation successful
0619 16:50:52.340033 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1742}
I20260619 16:50:52.341951 26378 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [term 1 FOLLOWER]: Refusing update from remote peer 21d4adb2e76f4fa7bd4a6cbbd84c353c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:52.342705 26527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:52.344126 26484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: Bootstrap starting.
I20260619 16:50:52.347280 26484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:52.348243 26484 log.cc:824] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:52.348901 26496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d4adb2e76f4fa7bd4a6cbbd84c353c. Latest consensus state: current_term: 1 leader_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.349058 26496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:52.349967 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.329261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.3.233.189:34847)
0619 16:50:52.329366 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.329370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.340285 (+ 10915us) server_negotiation.cc:407] Connection header received
0619 16:50:52.341138 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.341142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.341202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.341257 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.342209 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.342885 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.344096 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.344384 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.344458 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.346324 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.346328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.346331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.346357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.348103 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.348669 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.348676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.348679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.348779 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.349080 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.349084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.349086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.349315 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.349453 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.349686 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.349790 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:50:52.354177 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:52.328981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.3.233.188:42479)
0619 16:50:52.329102 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.329106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.335178 (+ 6072us) server_negotiation.cc:407] Connection header received
0619 16:50:52.335386 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.335389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.335451 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.335527 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.340267 (+ 4740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.341017 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.342617 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:52.342975 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:52.343053 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:52.350580 (+ 7527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:52.350585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:52.350588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:52.350615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:52.352342 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.352893 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.352898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.352900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.352984 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:52.353293 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:52.353296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:52.353298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:52.353476 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:52.353630 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:52.353862 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:50:52.353986 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:50:52.356040 26531 mvcc.cc:205] Tried to move back new op lower bound from 7298612643182899200 to 7298612643146297344. Current Snapshot: MvccSnapshot[applied={T|T < 7298612643182899200}]
I20260619 16:50:52.356397 26484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: No bootstrap required, opened a new log
I20260619 16:50:52.356431 26495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.356534 26495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:52.357369 26496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.357491 26496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:52.357892 26532 mvcc.cc:205] Tried to move back new op lower bound from 7298612643182899200 to 7298612643146297344. Current Snapshot: MvccSnapshot[applied={T|T < 7298612643182899200}]
I20260619 16:50:52.361217 26484 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.361447 26484 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec6947d68df4a5bac66f4ef5b066e9a, State: Initialized, Role: FOLLOWER
I20260619 16:50:52.361774 26484 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } }
I20260619 16:50:52.362370 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.362586 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:52.363025 26484 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:52.363067 26458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fec6947d68df4a5bac66f4ef5b066e9a"
I20260619 16:50:52.363287 26458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:52.364660 26458 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d4adb2e76f4fa7bd4a6cbbd84c353c in term 1.
I20260619 16:50:52.365597 26456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [term 1 FOLLOWER]: Refusing update from remote peer 21d4adb2e76f4fa7bd4a6cbbd84c353c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:52.366519 26527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:50:52.368629 26549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:52.368714 26549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:52.369311 26530 catalog_manager.cc:1347] Generated new cluster ID: cd7ee18bc3924902bd4a24e99d3012f7
I20260619 16:50:52.369370 26530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:52.372253 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d4adb2e76f4fa7bd4a6cbbd84c353c. Latest consensus state: current_term: 1 leader_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.372380 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:52.372473 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.372542 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:52.373334 26551 mvcc.cc:205] Tried to move back new op lower bound from 7298612643182899200 to 7298612643146297344. Current Snapshot: MvccSnapshot[applied={T|T < 7298612643182899200}]
I20260619 16:50:52.386894 26495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d4adb2e76f4fa7bd4a6cbbd84c353c" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 33301 } } peers { permanent_uuid: "3333eb9f5f6c4287ace8963b667c1c54" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34847 } } peers { permanent_uuid: "fec6947d68df4a5bac66f4ef5b066e9a" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 42479 } } }
I20260619 16:50:52.387066 26495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:52.392046 26530 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:52.399469 26530 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:52.409837 26530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21d4adb2e76f4fa7bd4a6cbbd84c353c: Generated new TSK 0
I20260619 16:50:52.410800 26530 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:50:52.435281 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.432114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.3.233.189:34847)
0619 16:50:52.432259 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.432263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.433930 (+ 1667us) server_negotiation.cc:407] Connection header received
0619 16:50:52.434372 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.434376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.434428 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.434482 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.435128 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35680: BlockingRecv error: recv got EOF from 127.0.0.1:35680 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 16:50:52.436717 26491 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.432963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.3.233.190:33301)
0619 16:50:52.433077 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.433082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.435773 (+ 2691us) server_negotiation.cc:407] Connection header received
0619 16:50:52.435951 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.435955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.436016 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.436067 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.436589 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54872: BlockingRecv error: recv got EOF from 127.0.0.1:54872 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 16:50:52.438125 26486 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.433583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.3.233.188:42479)
0619 16:50:52.433691 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.433696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.437109 (+ 3413us) server_negotiation.cc:407] Connection header received
0619 16:50:52.437381 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.437384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.437434 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.437514 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.438002 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36074: BlockingRecv error: recv got EOF from 127.0.0.1:36074 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 16:50:52.442839 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.440512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.3.233.189:34847)
0619 16:50:52.440614 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.440618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.441828 (+ 1210us) server_negotiation.cc:407] Connection header received
0619 16:50:52.442092 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.442096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.442176 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.442229 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.442732 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35692: BlockingRecv error: recv got EOF from 127.0.0.1:35692 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 16:50:52.447829 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.446233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.3.233.189:34847)
0619 16:50:52.446345 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.446349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.446872 (+ 523us) server_negotiation.cc:407] Connection header received
0619 16:50:52.447132 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.447136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.447186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.447237 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.447727 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35704: BlockingRecv error: recv got EOF from 127.0.0.1:35704 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 16:50:52.453192 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.451726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.3.233.189:34847)
0619 16:50:52.451826 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.451830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.452334 (+ 504us) server_negotiation.cc:407] Connection header received
0619 16:50:52.452508 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.452512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.452559 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.452611 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.453093 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35720: BlockingRecv error: recv got EOF from 127.0.0.1:35720 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 16:50:52.460985 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.459305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.3.233.189:34847)
0619 16:50:52.459444 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.459448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.459997 (+ 549us) server_negotiation.cc:407] Connection header received
0619 16:50:52.460186 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.460189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.460237 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.460300 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.460882 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35724: BlockingRecv error: recv got EOF from 127.0.0.1:35724 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 16:50:52.473091 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.471378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.3.233.189:34847)
0619 16:50:52.471508 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.471513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.472080 (+ 567us) server_negotiation.cc:407] Connection header received
0619 16:50:52.472298 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.472301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.472355 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.472407 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.472944 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40324: BlockingRecv error: recv got EOF from 127.0.0.1:40324 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 16:50:52.493556 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.491698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.3.233.189:34847)
0619 16:50:52.491830 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.491835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.492519 (+ 684us) server_negotiation.cc:407] Connection header received
0619 16:50:52.492724 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.492727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.492785 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.492837 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.493425 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40330: BlockingRecv error: recv got EOF from 127.0.0.1:40330 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 16:50:52.530194 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.528479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.3.233.189:34847)
0619 16:50:52.528601 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.528606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.529204 (+ 598us) server_negotiation.cc:407] Connection header received
0619 16:50:52.529412 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.529416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.529466 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.529518 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.530059 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40336: BlockingRecv error: recv got EOF from 127.0.0.1:40336 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 16:50:52.598101 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.596848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.3.233.189:34847)
0619 16:50:52.597029 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.597033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.597116 (+ 83us) server_negotiation.cc:407] Connection header received
0619 16:50:52.597263 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.597265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.597315 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.597392 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.597972 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40344: BlockingRecv error: recv got EOF from 127.0.0.1:40344 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 16:50:52.729835 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.728499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.3.233.189:34847)
0619 16:50:52.728778 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.728782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.728899 (+ 117us) server_negotiation.cc:407] Connection header received
0619 16:50:52.729040 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.729042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.729092 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.729178 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.729627 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40346: BlockingRecv error: recv got EOF from 127.0.0.1:40346 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
W20260619 16:50:52.989871 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:52.988627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.3.233.189:34847)
0619 16:50:52.988779 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:52.988783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:52.988836 (+ 53us) server_negotiation.cc:407] Connection header received
0619 16:50:52.989021 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:52.989024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:52.989075 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:52.989159 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:52.989737 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40356: BlockingRecv error: recv got EOF from 127.0.0.1:40356 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:50:53.316347 26518 catalog_manager.cc:1259] Loaded cluster ID: cd7ee18bc3924902bd4a24e99d3012f7
I20260619 16:50:53.316470 26518 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: loading cluster ID for follower catalog manager: success
I20260619 16:50:53.318212 26518 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: acquiring CA information for follower catalog manager: success
I20260619 16:50:53.319479 26518 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3333eb9f5f6c4287ace8963b667c1c54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:50:53.370287 26549 catalog_manager.cc:1259] Loaded cluster ID: cd7ee18bc3924902bd4a24e99d3012f7
I20260619 16:50:53.370404 26549 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: loading cluster ID for follower catalog manager: success
I20260619 16:50:53.372378 26549 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: acquiring CA information for follower catalog manager: success
I20260619 16:50:53.373638 26549 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fec6947d68df4a5bac66f4ef5b066e9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:50:53.508318 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:53.503982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.3.233.189:34847)
0619 16:50:53.504277 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:53.504288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:53.504309 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:50:53.504479 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:53.504482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:53.504614 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:53.504723 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:53.505002 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.505558 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.506527 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.506794 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.506901 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:53.507337 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:53.507820 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:53.507935 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:53.508040 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 16:50:53.508102 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260619 16:50:53.516938 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:53.512451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.3.233.188:42479)
0619 16:50:53.512717 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:53.512721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:53.512737 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:53.512849 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:53.512852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:53.512980 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:53.513093 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:53.513393 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.514106 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.515318 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.515552 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.515700 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:53.516034 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:53.516516 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:53.516691 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:53.516747 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:50:53.516806 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260619 16:50:53.525861 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:53.520450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.3.233.189:34847)
0619 16:50:53.520613 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:53.520618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:53.521238 (+ 620us) server_negotiation.cc:407] Connection header received
0619 16:50:53.521335 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:53.521339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:53.521400 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:53.521480 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:53.521948 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.522862 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.524437 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.524818 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.524923 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:53.525237 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:53.525337 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:53.525403 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:53.525563 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:50:53.525646 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:50:53.526335 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:53.520797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.3.233.188:42479)
0619 16:50:53.520923 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:53.520927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:53.521328 (+ 401us) server_negotiation.cc:407] Connection header received
0619 16:50:53.521483 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:53.521486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:53.521532 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:53.521617 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:53.522006 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.522784 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.525008 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.525298 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.525408 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:53.525791 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:53.525880 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:53.525949 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:53.526125 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 16:50:53.526190 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:50:53.526576 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:53.521019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.3.233.190:33301)
0619 16:50:53.521309 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:53.521314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:53.521326 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:50:53.521412 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:53.521416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:53.521589 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:53.521788 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:53.522959 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.523723 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.524925 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.525287 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.525393 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:53.525772 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:53.526194 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:53.526291 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:53.526361 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 16:50:53.526425 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
I20260619 16:50:53.530561 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26266
I20260619 16:50:53.537695 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26341
I20260619 16:50:53.543795 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26416
2026-06-19T16:50:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3121 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:53 dist-test-slave-cgwx krb5kdc[26638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:53 dist-test-slave-cgwx krb5kdc[26638](info): set up 2 sockets
Jun 19 16:50:53 dist-test-slave-cgwx krb5kdc[26638](info): commencing operation
krb5kdc: starting...
W20260619 16:50:54.593044 4006 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/test-user.keytab.
Jun 19 16:50:54 dist-test-slave-cgwx krb5kdc[26638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.233.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
Entry for principal kudu/127.3.233.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab.
I20260619 16:50:54.730515 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:45251
--webserver_interface=127.3.233.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:45251,127.3.233.189:34561,127.3.233.188:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:54.843465 26651 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:50:54.843775 26651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:54.843798 26651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:54.843892 26651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:54.848019 26651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:54.848083 26651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:54.848104 26651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:54.848120 26651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:54.848156 26651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:54.853471 26651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:45251,127.3.233.189:34561,127.3.233.188:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.190.keytab
--principal=kudu/127.3.233.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.190:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.233.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:54.854574 26651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:54.855669 26651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:54.864089 26657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:54.864063 26656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:54.864300 26651 server_base.cc:1034] running on GCE node
W20260619 16:50:54.864084 26659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:54.864907 26651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:54.866051 26651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:54.867236 26651 hybrid_clock.cc:648] HybridClock initialized: now 1781887854867215 us; error 36 us; skew 500 ppm
Jun 19 16:50:54 dist-test-slave-cgwx krb5kdc[26638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:54.870141 26651 init.cc:377] Logged in from keytab as kudu/127.3.233.190@KRBTEST.COM (short username kudu)
I20260619 16:50:54.871193 26651 webserver.cc:460] Webserver started at http://127.3.233.190:36007/ using document root <none> and password file <none>
I20260619 16:50:54.871757 26651 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:54.871994 26651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:54.872247 26651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:54.874629 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data/instance:
uuid: "5647f4b690c74d32952ec26c1570c220"
format_stamp: "Formatted at 2026-06-19 16:50:54 on dist-test-slave-cgwx"
server_key: "b9c4dce90322db2d999fcfc0f28340ad"
server_key_iv: "4f3c9d1d086a6d5ad8a90d4025c593b1"
server_key_version: "encryptionkey@0"
I20260619 16:50:54.875219 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/wal/instance:
uuid: "5647f4b690c74d32952ec26c1570c220"
format_stamp: "Formatted at 2026-06-19 16:50:54 on dist-test-slave-cgwx"
server_key: "b9c4dce90322db2d999fcfc0f28340ad"
server_key_iv: "4f3c9d1d086a6d5ad8a90d4025c593b1"
server_key_version: "encryptionkey@0"
I20260619 16:50:54.879026 26651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:50:54.881361 26666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:54.882681 26651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:50:54.882853 26651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/wal
uuid: "5647f4b690c74d32952ec26c1570c220"
format_stamp: "Formatted at 2026-06-19 16:50:54 on dist-test-slave-cgwx"
server_key: "b9c4dce90322db2d999fcfc0f28340ad"
server_key_iv: "4f3c9d1d086a6d5ad8a90d4025c593b1"
server_key_version: "encryptionkey@0"
I20260619 16:50:54.882995 26651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:54.904778 26651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:54.908064 26651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:54.908282 26651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:54.915462 26651 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.190:45251
I20260619 16:50:54.915797 26718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.190:45251 every 8 connection(s)
I20260619 16:50:54.916433 26651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:54.916878 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26651
I20260619 16:50:54.917026 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-0/wal/instance
I20260619 16:50:54.917315 4006 external_mini_cluster.cc:1428] Setting key 93eef6c32908f107b3b5e5ead8a96a87
I20260619 16:50:54.919657 26719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:54.923389 26719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.233.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.189@KRBTEST.COM" created.
Jun 19 16:50:54 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:54.939975 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:54.925661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:45251 (local address 127.0.0.1:47754)
0619 16:50:54.926132 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:54.926180 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:54.927187 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:54.927907 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:54.927915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:54.928424 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:54.928824 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:54.928839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:54.930432 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:54.930447 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:54.931842 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:54.931857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:54.932529 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:54.932534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:54.932710 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:54.933360 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:54.933384 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:54.935053 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:54.937009 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:54.937017 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:54.937025 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:54.937347 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:54.937656 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:54.937658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:54.937660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:54.937794 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:54.938484 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:54.938489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:54.938677 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 16:50:54.938856 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:50:54.939253 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":77,"thread_start_us":138,"threads_started":1}
I20260619 16:50:54.939986 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:54.925661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.3.233.190:45251)
0619 16:50:54.926080 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:54.926094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:54.926286 (+ 192us) server_negotiation.cc:407] Connection header received
0619 16:50:54.927492 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:54.927497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:54.927686 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:54.927848 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:54.928987 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:54.930296 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:54.931995 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:54.932355 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:54.932476 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:54.935206 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:54.935215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:54.935230 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:54.935258 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:54.936856 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:54.937457 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:54.937462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:54.937464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:54.937570 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:54.937903 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:54.937908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:54.937913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:54.938385 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:54.938491 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:54.938971 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 16:50:54.939158 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":27520,"thread_start_us":147,"threads_started":1}
I20260619 16:50:54.942471 26719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:54.943286 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:54.943041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35564)
0619 16:50:54.943148 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:54.943222 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab.
Entry for principal kudu/127.3.233.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab.
W20260619 16:50:54.943554 26668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
I20260619 16:50:54.944449 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:34561
--webserver_interface=127.3.233.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab
--principal=kudu/127.3.233.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:45251,127.3.233.189:34561,127.3.233.188:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:50:54.944538 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:54.944301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35576)
0619 16:50:54.944413 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:54.944473 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:50:54.944788 26719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:34561: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
I20260619 16:50:54.994052 26719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } attempt: 1
I20260619 16:50:54.995143 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:54.994785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35582)
0619 16:50:54.994971 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:54.995043 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:50:54.996059 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:54.995781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35584)
0619 16:50:54.995933 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:54.995986 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 16:50:54.996232 26719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:34561: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
W20260619 16:50:55.058454 26726 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:50:55.058738 26726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:55.058761 26726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:55.058841 26726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:55.063117 26726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:55.063174 26726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:55.063194 26726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:55.063210 26726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:55.063254 26726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:55.064415 26719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } attempt: 2
I20260619 16:50:55.065220 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.064933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35592)
0619 16:50:55.065080 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.065140 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:50:55.065966 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.065732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35596)
0619 16:50:55.065857 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.065907 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:50:55.066128 26719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.189:34561: Network error: Client connection negotiation failed: client connection to 127.3.233.189:34561: connect: Connection refused (error 111)
I20260619 16:50:55.069186 26726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:45251,127.3.233.189:34561,127.3.233.188:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.189.keytab
--principal=kudu/127.3.233.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.189:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.233.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:55.070508 26726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:55.071652 26726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:55.080055 26731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.080071 26732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.080142 26734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:55.080125 26726 server_base.cc:1034] running on GCE node
I20260619 16:50:55.080821 26726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:55.081986 26726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:55.083170 26726 hybrid_clock.cc:648] HybridClock initialized: now 1781887855083154 us; error 56 us; skew 500 ppm
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:55.086169 26726 init.cc:377] Logged in from keytab as kudu/127.3.233.189@KRBTEST.COM (short username kudu)
I20260619 16:50:55.087251 26726 webserver.cc:460] Webserver started at http://127.3.233.189:42007/ using document root <none> and password file <none>
I20260619 16:50:55.087817 26726 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:55.087883 26726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:55.088052 26726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:55.090061 26726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data/instance:
uuid: "f1b468bd42f34fc28dd0deea31f3757b"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-cgwx"
server_key: "facc31298fddb9a7450001c7097e736f"
server_key_iv: "3d69636e4b767152d1b7fb459ca19da0"
server_key_version: "encryptionkey@0"
I20260619 16:50:55.090574 26726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/wal/instance:
uuid: "f1b468bd42f34fc28dd0deea31f3757b"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-cgwx"
server_key: "facc31298fddb9a7450001c7097e736f"
server_key_iv: "3d69636e4b767152d1b7fb459ca19da0"
server_key_version: "encryptionkey@0"
I20260619 16:50:55.094321 26726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:50:55.096952 26741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.098264 26726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:50:55.098407 26726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/wal
uuid: "f1b468bd42f34fc28dd0deea31f3757b"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-cgwx"
server_key: "facc31298fddb9a7450001c7097e736f"
server_key_iv: "3d69636e4b767152d1b7fb459ca19da0"
server_key_version: "encryptionkey@0"
I20260619 16:50:55.098518 26726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:55.117512 26726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:55.120945 26726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:55.121146 26726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:55.128008 26726 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.189:34561
I20260619 16:50:55.128293 26793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.189:34561 every 8 connection(s)
I20260619 16:50:55.129000 26726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/data/info.pb
I20260619 16:50:55.130892 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26726
I20260619 16:50:55.131049 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-1/wal/instance
I20260619 16:50:55.131342 4006 external_mini_cluster.cc:1428] Setting key d0e61b03a5f7938d6f2a2bed23545945
I20260619 16:50:55.132028 26794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:55.136423 26794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.233.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.233.188@KRBTEST.COM" created.
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:55.151320 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.138354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.3.233.190:45251)
0619 16:50:55.138483 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.138486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.139636 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 16:50:55.140684 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.140688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.140757 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.140843 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.142309 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.142882 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.144121 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.144377 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.144490 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.147009 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.147012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.147014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.147039 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.149099 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.149648 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.149653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.149655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.149772 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.150116 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.150122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.150123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.150340 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.150440 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.151003 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.151139 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:50:55.152127 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.138909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:45251 (local address 127.0.0.1:47758)
0619 16:50:55.139479 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.139523 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.140458 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.141026 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.141033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.141458 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.142149 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.142171 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.143092 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.143095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.143993 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.144002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.144530 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.144535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.144702 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.145348 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.145364 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.146794 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.149241 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.149248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.149265 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.149540 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.149864 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.149868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.149870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.150018 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.150437 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.150445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.150716 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.150907 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.151291 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":180,"threads_started":1}
I20260619 16:50:55.153715 26794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.233.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab.
Entry for principal kudu/127.3.233.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab.
I20260619 16:50:55.157292 26719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } attempt: 3
I20260619 16:50:55.158308 4006 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
/tmp/dist-test-taskfHMdDR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:45837
--webserver_interface=127.3.233.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab
--principal=kudu/127.3.233.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.233.148:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.233.190:45251,127.3.233.189:34561,127.3.233.188:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:50:55.166808 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.154267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35604)
0619 16:50:55.154419 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.154436 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.154524 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.155172 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.155175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.155334 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.155435 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.155440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.156525 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.156528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.157138 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.157147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.158094 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.158100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.158251 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.158855 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.158876 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.160382 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.164210 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.164216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.164221 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.164544 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.165312 (+ 768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.165318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.165322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.165456 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.166047 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.166053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.166176 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.166302 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.166509 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:50:55.166862 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.154407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.3.233.189:34561)
0619 16:50:55.154726 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.154736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.154754 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:50:55.154805 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.154809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.155035 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.155182 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.155549 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.156422 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.157250 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.157834 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.157939 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.160529 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.160540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.160562 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.160582 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.164094 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.164668 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.164677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.164680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.165210 (+ 530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.165573 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.165579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.165583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.165922 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.166072 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.166416 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.166620 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":165,"threads_started":1}
I20260619 16:50:55.168331 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.157762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35606)
0619 16:50:55.159069 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.159082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.159217 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.159487 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.159490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.159676 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.159796 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.159802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.160955 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.160960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.161545 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.161554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.162238 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.162241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.162326 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.162751 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.162763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.164184 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.166769 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.166772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.166774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.167049 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.167360 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.167362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.167364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.167461 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.167807 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.167810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.167913 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.168013 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.168166 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219}
I20260619 16:50:55.168339 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.157811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.3.233.189:34561)
0619 16:50:55.158106 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.158116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.159142 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 16:50:55.159322 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.159325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.159381 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.159469 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.159903 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.160771 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.161661 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.162102 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.162213 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.164308 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.164311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.164313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.164333 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.166631 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.167150 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.167152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.167154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.167251 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.167563 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.167565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.167566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.167716 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.167815 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.168079 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.168183 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":146,"threads_started":1}
I20260619 16:50:55.169343 26794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:55.169550 26719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:55.170173 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.169863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43898)
0619 16:50:55.169993 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.170074 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:50:55.170295 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.169997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43906)
0619 16:50:55.170137 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.170198 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:50:55.170593 26745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
I20260619 16:50:55.171114 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.170844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43914)
0619 16:50:55.171006 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.171055 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 16:50:55.171309 26719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:45837: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
I20260619 16:50:55.171679 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.171503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43926)
0619 16:50:55.171592 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.171634 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 16:50:55.171818 26794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:45837: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
I20260619 16:50:55.202527 26719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } attempt: 1
I20260619 16:50:55.203478 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.203194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43936)
0619 16:50:55.203329 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.203400 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:50:55.204236 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.203997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43940)
0619 16:50:55.204133 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.204178 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:50:55.204365 26719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:45837: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
I20260619 16:50:55.221074 26794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } attempt: 1
I20260619 16:50:55.221905 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.221623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43946)
0619 16:50:55.221762 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.221830 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:50:55.222702 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.222427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43960)
0619 16:50:55.222536 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.222589 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:50:55.222837 26794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:45837: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
W20260619 16:50:55.274379 26801 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:50:55.274776 26801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:55.274837 26801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:55.274915 26801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:55.279011 26801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:55.279081 26801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:55.279115 26801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:55.279151 26801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:55.279193 26801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:55.279518 26719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } attempt: 2
I20260619 16:50:55.280381 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.280033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43962)
0619 16:50:55.280225 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.280297 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:50:55.281144 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.280908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43966)
0619 16:50:55.281026 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.281088 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:50:55.281273 26719 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:45837: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
I20260619 16:50:55.284904 26801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.233.148:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.233.190:45251,127.3.233.189:34561,127.3.233.188:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/krb5kdc/kudu_127.3.233.188.keytab
--principal=kudu/127.3.233.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.233.188:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.233.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:50:55.286006 26801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:55.287146 26801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:50:55.290987 26794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } attempt: 2
I20260619 16:50:55.291879 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.291479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43968)
0619 16:50:55.291717 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.291791 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:50:55.292605 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.292367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43976)
0619 16:50:55.292482 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.292535 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:50:55.292796 26794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.233.188:45837: Network error: Client connection negotiation failed: client connection to 127.3.233.188:45837: connect: Connection refused (error 111)
W20260619 16:50:55.295862 26810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.295908 26807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.295897 26808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:55.296051 26801 server_base.cc:1034] running on GCE node
I20260619 16:50:55.296644 26801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:55.297804 26801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:55.298996 26801 hybrid_clock.cc:648] HybridClock initialized: now 1781887855298946 us; error 63 us; skew 500 ppm
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:55.302023 26801 init.cc:377] Logged in from keytab as kudu/127.3.233.188@KRBTEST.COM (short username kudu)
I20260619 16:50:55.303059 26801 webserver.cc:460] Webserver started at http://127.3.233.188:37993/ using document root <none> and password file <none>
I20260619 16:50:55.303637 26801 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:55.303728 26801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:55.303951 26801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:55.306027 26801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data/instance:
uuid: "6dd1bb898e87478b8aa2113c9f462928"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-cgwx"
server_key: "c3f19574adaa679d4091ab3c2f73cd1c"
server_key_iv: "75491a7b87a6aac53395e2db2c1efbe2"
server_key_version: "encryptionkey@0"
I20260619 16:50:55.306613 26801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/wal/instance:
uuid: "6dd1bb898e87478b8aa2113c9f462928"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-cgwx"
server_key: "c3f19574adaa679d4091ab3c2f73cd1c"
server_key_iv: "75491a7b87a6aac53395e2db2c1efbe2"
server_key_version: "encryptionkey@0"
I20260619 16:50:55.310572 26801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 16:50:55.313105 26817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.314417 26801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:50:55.314522 26801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data,/tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/wal
uuid: "6dd1bb898e87478b8aa2113c9f462928"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-cgwx"
server_key: "c3f19574adaa679d4091ab3c2f73cd1c"
server_key_iv: "75491a7b87a6aac53395e2db2c1efbe2"
server_key_version: "encryptionkey@0"
I20260619 16:50:55.314610 26801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:55.344867 26801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:55.348305 26801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:55.348469 26801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:55.355720 26801 rpc_server.cc:307] RPC server started. Bound to: 127.3.233.188:45837
I20260619 16:50:55.356014 26869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.233.188:45837 every 8 connection(s)
I20260619 16:50:55.356755 26801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/data/info.pb
I20260619 16:50:55.359712 26870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:55.364450 26870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:55.366158 4006 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu as pid 26801
I20260619 16:50:55.366303 4006 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfHMdDR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887569513258-4006-0/minicluster-data/master-2/wal/instance
I20260619 16:50:55.366612 4006 external_mini_cluster.cc:1428] Setting key e9dbbf5e87804db76abb81160559e736
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:50:55.378355 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.366282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.3.233.190:45251)
0619 16:50:55.366434 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.366439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.367398 (+ 959us) server_negotiation.cc:407] Connection header received
0619 16:50:55.368527 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.368530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.368574 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.368639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.369562 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.370058 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.371226 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.371569 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.371645 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.374202 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.374206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.374209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.374241 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.375997 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.376491 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.376495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.376497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.376586 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.377024 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.377026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.377027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.377249 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.377356 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.378039 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.378166 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:50:55.379277 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.366736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.190:45251 (local address 127.0.0.1:47768)
0619 16:50:55.367247 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.367299 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.368316 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.368704 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.368716 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.369145 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.369406 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.369439 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.370163 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.370166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.371125 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.371134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.371713 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.371716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.371912 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.372618 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.372633 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.374061 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.376114 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.376122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.376140 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.376390 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.376713 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.376716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.376717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.376915 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.377412 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.377420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.377792 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.377975 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.378314 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":175,"threads_started":1}
I20260619 16:50:55.379283 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.367268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.3.233.188:45837)
0619 16:50:55.367755 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.367765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.367802 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:50:55.368316 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.368339 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.368585 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.368774 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.369493 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.370572 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.371544 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.372098 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.372248 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.375019 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.375028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.375046 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.375077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.376758 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.377249 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.377256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.377257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.377348 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.377645 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.377647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.377648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.377881 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.378095 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.378327 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.378637 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":132,"threads_started":1}
I20260619 16:50:55.380426 26719 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } attempt: 3
I20260619 16:50:55.380584 26870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:55.383963 26794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } attempt: 3
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.190@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.190@KRBTEST.COM
I20260619 16:50:55.391409 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.380808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43990)
0619 16:50:55.380945 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.380962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.381033 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.381590 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.381593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.381728 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.381805 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.381810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.382715 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.382720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.383290 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.383298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.383775 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.383780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.383866 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.384414 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.384427 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.385988 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.389422 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.389427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.389431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.389712 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.389989 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.390236 (+ 247us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.390239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.390326 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.390787 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.390792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.390899 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.391052 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.391207 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":36}
I20260619 16:50:55.391422 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.380862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.3.233.188:45837)
0619 16:50:55.381187 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.381191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.381211 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:50:55.381242 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.381244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.381283 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.381392 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.381952 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.382614 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.383394 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.383672 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.383798 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.386135 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.386139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.386142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.386176 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.389318 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.389799 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.389802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.389804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.389886 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.390430 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.390433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.390435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.390684 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.390777 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.391113 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.391244 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":238,"spinlock_wait_cycles":7936}
I20260619 16:50:55.391590 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.381501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.3.233.189:34561)
0619 16:50:55.381643 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.381647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.381662 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:50:55.381715 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.381718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.381766 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.381852 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.382252 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.382805 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385051 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.385450 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385554 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.387938 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.387944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.387949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.387978 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.389736 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.390213 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.390218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.390221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.390317 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.390677 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.390680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.390682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.390890 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.391010 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.391332 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.391448 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.189@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:50:55.392030 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.381040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.189:34561 (local address 127.0.0.1:35622)
0619 16:50:55.381157 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.381186 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.381258 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.381885 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.381890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.382053 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.382151 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.382157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.382911 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.382917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.383835 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.383847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.385556 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.385666 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.386156 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.386175 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.387767 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.389854 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.389857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.389859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.390096 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.390430 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.390433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.390435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.390559 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.391078 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.391081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.391155 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.391738 (+ 583us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.391867 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":25,"spinlock_wait_cycles":2048}
I20260619 16:50:55.395638 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.384635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43992)
0619 16:50:55.384860 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.384875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.384959 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.386636 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.386641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.386861 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.386983 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.386990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.388094 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.388100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.388995 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.389008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.390097 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.390102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.390218 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.390646 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.390659 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.392340 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.394057 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.394060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.394062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.394331 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.394617 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.394620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.394622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.394733 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.395179 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.395185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.395286 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.395364 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.395485 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 16:50:55.395641 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.381888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.3.233.190:45251)
0619 16:50:55.381979 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.381983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.382131 (+ 148us) server_negotiation.cc:407] Connection header received
0619 16:50:55.382973 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.382976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.383017 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.383084 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.383864 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.384449 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385349 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.385622 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385837 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.389751 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.389757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.389763 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.389794 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.393781 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.394346 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.394350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.394352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.394446 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.394745 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.394749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.394753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.394993 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.395229 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.395360 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.395468 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":38}
I20260619 16:50:55.395967 26719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220: Bootstrap starting.
I20260619 16:50:55.396584 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.384872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.3.233.188:45837)
0619 16:50:55.386388 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.386400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.386414 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:50:55.386446 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.386450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.386492 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.386599 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.387089 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.387966 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.389153 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.389985 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.390086 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.392479 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.392482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.392485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.392506 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.393929 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.394433 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.394439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.394441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.394521 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.394815 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.394819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.394820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.395062 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.396228 (+ 1166us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.396304 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.396412 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":116,"threads_started":1}
I20260619 16:50:55.398398 26719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:55.399184 26719 log.cc:824] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:55.400213 26870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:50:55.401619 26719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220: No bootstrap required, opened a new log
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.233.189@KRBTEST.COM
I20260619 16:50:55.402693 26794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: Bootstrap starting.
I20260619 16:50:55.405097 26794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:55.405838 26794 log.cc:824] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:55.406229 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.396685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.3.233.189:34561)
0619 16:50:55.396897 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.396902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.396914 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:50:55.397009 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.397012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.397050 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.397148 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.397718 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.398202 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.399034 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.399286 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.399374 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.402313 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.402317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.402319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.402344 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.404107 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.404587 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.404591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.404593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.404771 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.405136 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.405140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.405142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.405393 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.405562 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.405961 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.406083 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 16:50:55.406476 26719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
I20260619 16:50:55.406740 26719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5647f4b690c74d32952ec26c1570c220, State: Initialized, Role: FOLLOWER
I20260619 16:50:55.407140 26719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
I20260619 16:50:55.408488 26794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: No bootstrap required, opened a new log
I20260619 16:50:55.408615 26719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:50:55.409770 26672 tablet.cc:2369] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:55.410079 4006 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 19 16:50:55 dist-test-slave-cgwx krb5kdc[26638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.233.188@KRBTEST.COM for kudu/127.3.233.188@KRBTEST.COM
I20260619 16:50:55.411371 26693 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5647f4b690c74d32952ec26c1570c220"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47778
I20260619 16:50:55.412397 26693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:50:55.412518 26693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:55.413584 26794 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
I20260619 16:50:55.413816 26794 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b468bd42f34fc28dd0deea31f3757b, State: Initialized, Role: FOLLOWER
W20260619 16:50:55.414211 26889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 16:50:55.414149 26794 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
I20260619 16:50:55.414270 26693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
I20260619 16:50:55.415555 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.400664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.233.188:45837 (local address 127.0.0.1:43996)
0619 16:50:55.400941 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.400955 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.401021 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.406704 (+ 5683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.406707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.406894 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.407040 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.407048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.407997 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.408000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.408736 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.408747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.409380 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.409383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.409453 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.410295 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.410309 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.412071 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.414064 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.414067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.414069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.414307 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.414589 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.414591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.414593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.414662 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.415081 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.415083 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.415162 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.415273 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.415400 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260619 16:50:55.415575 26794 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:55.416136 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.400852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.3.233.188:45837)
0619 16:50:55.406371 (+ 5519us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.406377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.406393 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:50:55.406432 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.406435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.406479 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.406579 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.407162 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.407885 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.408860 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.409167 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.409273 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.412202 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.412215 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.412217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.412240 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.413957 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.414416 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.414422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.414426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.414501 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.414741 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.414744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.414745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.414956 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.415045 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.415886 (+ 841us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.416003 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5409}
I20260619 16:50:55.419265 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: Bootstrap starting.
I20260619 16:50:55.417637 26651 master_runner.cc:186] Error getting master registration for 127.3.233.189:34561: OK, instance_id { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" instance_seqno: 1781887855126060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:50:55.421022 26651 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:50:55.421886 26870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:55.422729 26870 log.cc:824] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:55.422994 26693 leader_election.cc:288] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [CANDIDATE]: Term 1 election: Requested vote from peers f1b468bd42f34fc28dd0deea31f3757b (127.3.233.189:34561), 6dd1bb898e87478b8aa2113c9f462928 (127.3.233.188:45837)
I20260619 16:50:55.424440 26878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.424588 26878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:50:55.414273 26889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 16:50:55.421296 26902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 16:50:55.422752 26767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5647f4b690c74d32952ec26c1570c220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f1b468bd42f34fc28dd0deea31f3757b"
I20260619 16:50:55.426198 26767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:55.427582 26767 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5647f4b690c74d32952ec26c1570c220 in term 1.
I20260619 16:50:55.428043 26668 leader_election.cc:302] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5647f4b690c74d32952ec26c1570c220, f1b468bd42f34fc28dd0deea31f3757b; no voters:
I20260619 16:50:55.428141 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: No bootstrap required, opened a new log
I20260619 16:50:55.428349 26878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:55.428580 26878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [term 1 LEADER]: Becoming Leader. State: Replica: 5647f4b690c74d32952ec26c1570c220, State: Running, Role: LEADER
I20260619 16:50:55.428915 26878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
W20260619 16:50:55.429224 26902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:55.429416 26891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.429513 26891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:55.432462 26870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
I20260619 16:50:55.432695 26870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd1bb898e87478b8aa2113c9f462928, State: Initialized, Role: FOLLOWER
I20260619 16:50:55.433009 26870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } }
I20260619 16:50:55.429934 26726 master_runner.cc:186] Error getting master registration for 127.3.233.188:45837: OK, instance_id { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" instance_seqno: 1781887855353637 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:50:55.433326 26726 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:50:55.434408 26870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:55.434489 26844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5647f4b690c74d32952ec26c1570c220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6dd1bb898e87478b8aa2113c9f462928"
I20260619 16:50:55.434736 26844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:55.436364 26844 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5647f4b690c74d32952ec26c1570c220 in term 1.
I20260619 16:50:55.434278 26912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5647f4b690c74d32952ec26c1570c220. Latest consensus state: current_term: 1 leader_uuid: "5647f4b690c74d32952ec26c1570c220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.437224 26912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: This master's current role is: LEADER
W20260619 16:50:55.440026 26925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:50:55.440110 26925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:50:55.440461 26926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:55.440449 26914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.440562 26914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:55.441534 26926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:55.447000 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.425754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.3.233.189:34561)
0619 16:50:55.425893 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.425897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.429804 (+ 3907us) server_negotiation.cc:407] Connection header received
0619 16:50:55.429980 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.429983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.430030 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.430081 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.434304 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.435360 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.436650 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.437016 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.437096 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.441036 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.441043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.441049 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.441078 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.444890 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.445537 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.445542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.445544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.445653 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.445942 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.445945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.445947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.446180 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.446303 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.446670 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.446794 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:50:55.449234 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.426401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.3.233.188:45837)
0619 16:50:55.426523 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.426528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.435652 (+ 9124us) server_negotiation.cc:407] Connection header received
0619 16:50:55.435911 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.435914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.435964 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.436019 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.441362 (+ 5343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.442077 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.443305 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.443608 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.443682 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.445485 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.445490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.445493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.445518 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.447260 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.447813 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.447817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.447819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.447902 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.448196 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.448199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.448201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.448415 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.448552 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.448892 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.449010 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:50:55.453435 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.425340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.3.233.190:45251)
0619 16:50:55.425465 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.425471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.435445 (+ 9974us) server_negotiation.cc:407] Connection header received
0619 16:50:55.436169 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.436172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.436234 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.436294 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.439515 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.440208 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.447907 (+ 7699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.448214 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.448282 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.449936 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.449940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.449943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.449972 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.451703 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.452179 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.452183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.452185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.452268 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.452546 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.452549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.452551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.452771 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.452894 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.453143 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.453262 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:50:55.458629 26844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [term 1 FOLLOWER]: Refusing update from remote peer 5647f4b690c74d32952ec26c1570c220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:55.458842 26767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [term 1 FOLLOWER]: Refusing update from remote peer 5647f4b690c74d32952ec26c1570c220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:55.459444 26878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:55.459896 26878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:55.464003 26891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5647f4b690c74d32952ec26c1570c220. Latest consensus state: current_term: 1 leader_uuid: "5647f4b690c74d32952ec26c1570c220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.464156 26891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:55.468261 26914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5647f4b690c74d32952ec26c1570c220. Latest consensus state: current_term: 1 leader_uuid: "5647f4b690c74d32952ec26c1570c220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.468389 26914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:55.472155 26912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5647f4b690c74d32952ec26c1570c220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.472299 26912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:55.472410 26928 mvcc.cc:205] Tried to move back new op lower bound from 7298612655951106048 to 7298612655840251904. Current Snapshot: MvccSnapshot[applied={T|T < 7298612655951106048}]
I20260619 16:50:55.473567 26914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5647f4b690c74d32952ec26c1570c220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.473695 26914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:55.474644 26878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5647f4b690c74d32952ec26c1570c220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.474822 26878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:55.475989 26891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5647f4b690c74d32952ec26c1570c220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5647f4b690c74d32952ec26c1570c220" member_type: VOTER last_known_addr { host: "127.3.233.190" port: 45251 } } peers { permanent_uuid: "f1b468bd42f34fc28dd0deea31f3757b" member_type: VOTER last_known_addr { host: "127.3.233.189" port: 34561 } } peers { permanent_uuid: "6dd1bb898e87478b8aa2113c9f462928" member_type: VOTER last_known_addr { host: "127.3.233.188" port: 45837 } } }
I20260619 16:50:55.476132 26891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:50:55.476315 26929 mvcc.cc:205] Tried to move back new op lower bound from 7298612655951106048 to 7298612655840251904. Current Snapshot: MvccSnapshot[applied={T|T < 7298612655951106048}]
I20260619 16:50:55.479157 26926 catalog_manager.cc:1347] Generated new cluster ID: d677b605fccb4eb4a39a2ff5e41ef612
I20260619 16:50:55.479228 26926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:55.481081 26931 mvcc.cc:205] Tried to move back new op lower bound from 7298612655951106048 to 7298612655840251904. Current Snapshot: MvccSnapshot[applied={T|T < 7298612655951106048}]
I20260619 16:50:55.512817 26926 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:55.514262 26926 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:55.528775 26926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5647f4b690c74d32952ec26c1570c220: Generated new TSK 0
I20260619 16:50:55.529814 26926 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:50:55.563114 26875 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.560428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.3.233.188:45837)
0619 16:50:55.560566 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.560571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.561936 (+ 1365us) server_negotiation.cc:407] Connection header received
0619 16:50:55.562139 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.562143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.562195 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.562262 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.562952 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44016: BlockingRecv error: recv got EOF from 127.0.0.1:44016 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:50:55.564608 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.561320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.3.233.189:34561)
0619 16:50:55.561438 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.561443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.563635 (+ 2192us) server_negotiation.cc:407] Connection header received
0619 16:50:55.563821 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.563824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.563877 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.563930 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.564486 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35662: BlockingRecv error: recv got EOF from 127.0.0.1:35662 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:50:55.566653 26723 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.565106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47792 (local address 127.3.233.190:45251)
0619 16:50:55.565217 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.565223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.565614 (+ 391us) server_negotiation.cc:407] Connection header received
0619 16:50:55.565802 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.565806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.565861 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.565919 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.566451 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47792: BlockingRecv error: recv got EOF from 127.0.0.1:47792 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 16:50:55.571547 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.569841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.3.233.189:34561)
0619 16:50:55.569958 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.569963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.570459 (+ 496us) server_negotiation.cc:407] Connection header received
0619 16:50:55.570707 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.570711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.570795 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.570853 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.571435 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35678: BlockingRecv error: recv got EOF from 127.0.0.1:35678 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 16:50:55.576898 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.575261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.3.233.189:34561)
0619 16:50:55.575370 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.575375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.575920 (+ 545us) server_negotiation.cc:407] Connection header received
0619 16:50:55.576124 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.576127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.576183 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.576254 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.576791 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35692: BlockingRecv error: recv got EOF from 127.0.0.1:35692 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 16:50:55.583122 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.581307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.3.233.189:34561)
0619 16:50:55.581417 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.581422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.582012 (+ 590us) server_negotiation.cc:407] Connection header received
0619 16:50:55.582237 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.582241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.582301 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.582356 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.582912 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35708: BlockingRecv error: recv got EOF from 127.0.0.1:35708 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 16:50:55.591821 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.589850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.3.233.189:34561)
0619 16:50:55.590014 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.590019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.590625 (+ 606us) server_negotiation.cc:407] Connection header received
0619 16:50:55.590850 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.590854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.590944 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.591014 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.591696 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35712: BlockingRecv error: recv got EOF from 127.0.0.1:35712 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 16:50:55.605036 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.603104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.3.233.189:34561)
0619 16:50:55.603252 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.603257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.603895 (+ 638us) server_negotiation.cc:407] Connection header received
0619 16:50:55.604137 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.604141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.604219 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.604285 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.604906 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35722: BlockingRecv error: recv got EOF from 127.0.0.1:35722 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 16:50:55.625761 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.623968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.3.233.189:34561)
0619 16:50:55.624093 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.624098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.624697 (+ 599us) server_negotiation.cc:407] Connection header received
0619 16:50:55.624915 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.624919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.624977 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.625034 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.625626 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35734: BlockingRecv error: recv got EOF from 127.0.0.1:35734 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 16:50:55.661806 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.660365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.3.233.189:34561)
0619 16:50:55.660501 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.660505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.660862 (+ 357us) server_negotiation.cc:407] Connection header received
0619 16:50:55.661031 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.661035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.661096 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.661187 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.661663 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35744: BlockingRecv error: recv got EOF from 127.0.0.1:35744 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:50:55.729465 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.728190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.3.233.189:34561)
0619 16:50:55.728374 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.728379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.728527 (+ 148us) server_negotiation.cc:407] Connection header received
0619 16:50:55.728722 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.728725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.728781 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.728884 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.729361 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35758: BlockingRecv error: recv got EOF from 127.0.0.1:35758 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 16:50:55.860874 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:55.859639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.3.233.189:34561)
0619 16:50:55.859800 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.859805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.859939 (+ 134us) server_negotiation.cc:407] Connection header received
0619 16:50:55.860110 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.860113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.860196 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.860293 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.860746 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35772: BlockingRecv error: recv got EOF from 127.0.0.1:35772 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 16:50:56.120467 26800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:50:56.119304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.3.233.189:34561)
0619 16:50:56.119446 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.119450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.119589 (+ 139us) server_negotiation.cc:407] Connection header received
0619 16:50:56.119663 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.119666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.119720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.119807 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:56.120347 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35786: BlockingRecv error: recv got EOF from 127.0.0.1:35786 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:50:56.430588 26902 catalog_manager.cc:1259] Loaded cluster ID: d677b605fccb4eb4a39a2ff5e41ef612
I20260619 16:50:56.430691 26902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: loading cluster ID for follower catalog manager: success
I20260619 16:50:56.432415 26902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: acquiring CA information for follower catalog manager: success
I20260619 16:50:56.433583 26902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1b468bd42f34fc28dd0deea31f3757b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:50:56.441268 26925 catalog_manager.cc:1259] Loaded cluster ID: d677b605fccb4eb4a39a2ff5e41ef612
I20260619 16:50:56.441339 26925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: loading cluster ID for follower catalog manager: success
I20260619 16:50:56.442894 26925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: acquiring CA information for follower catalog manager: success
I20260619 16:50:56.444098 26925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6dd1bb898e87478b8aa2113c9f462928: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:50:56.638687 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.634703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.3.233.189:34561)
0619 16:50:56.634962 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.634966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.634984 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:50:56.635072 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.635075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.635204 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.635376 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:56.635674 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.636209 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.637179 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.637418 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.637527 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.637853 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:56.638289 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:56.638425 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.638468 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.638527 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260619 16:50:56.645478 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.641852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.3.233.188:45837)
0619 16:50:56.642100 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.642104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.642118 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:50:56.642202 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.642214 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.642336 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.642443 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:56.642615 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.643244 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.644031 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.644261 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.644340 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.644580 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:56.645014 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:56.645109 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.645239 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.645302 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260619 16:50:56.652386 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.649116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.3.233.189:34561)
0619 16:50:56.649246 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.649250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.649275 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:50:56.649331 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.649336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.649406 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.649486 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:56.649845 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.650474 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.651353 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.651569 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.651674 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.651910 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:56.651979 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:56.652054 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.652161 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.652221 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:50:56.653893 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.649468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.3.233.188:45837)
0619 16:50:56.649568 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.649572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.649773 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:50:56.649906 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.649911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.649968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.650037 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:56.650403 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.651227 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.652480 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.652833 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.652916 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.653482 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:56.653557 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:56.653630 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.653731 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.653783 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:50:56.654536 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.648919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.3.233.190:45251)
0619 16:50:56.649181 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.649188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.649518 (+ 330us) server_negotiation.cc:407] Connection header received
0619 16:50:56.649622 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.649626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.649821 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.649954 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:56.650133 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.651073 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.652436 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.652834 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.652929 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.653540 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:56.653964 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:56.654257 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.654321 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.654382 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":66,"threads_started":1}
I20260619 16:50:56.657464 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26651
I20260619 16:50:56.664075 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26726
I20260619 16:50:56.670239 4006 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfHMdDR/build/debug/bin/kudu with pid 26801
2026-06-19T16:50:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3126 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6248 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287149 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST